Troubleshooting Guide

Introduction

This guide provides comprehensive information on how to install and configure the Nexthink CMDB Populator integration with ServiceNow, as well as basic maintenance guides. This document provides a detailed description of processes needed for a successful installation.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us via email [e-mail].

This document is intended for readers with a detailed understanding of the Nexthink technology and the ServiceNow technology.

These configuration instructions must be executed by a ServiceNow certified professional.

For more information on Nexthink technology you can access the entire technical documentation at: http://doc.nexthink.com/Documentation

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

Revision History

Date	History
2021-04-06	Added section “5.5 Not trusted certificates in Quebec release“
2020-09-18	Added IRE troubleshooting
2019-12-18	Added Cloud deployments troubleshooting
2019-10-11	Added field normalization support

Quick resolution tips

General Query-Related errors

Invalid username/password combo (HTTP 401/403)	Edit the Authentication Profiles used for the Engine and set the proper user and password
Nexthink Categories missing (HTTP 400)	Make sure the categories have been correctly imported into the Finder
Field '<field_name>' of table 'device' is not defined for platform '<platform>'	Limit the fields to be imported to those existing for all desired platforms or select only the appropriate platforms by checking/unchecking the corresponding ServiceNow properties
No data is imported from Engine	Make sure the Finder Categories have been properly modified to select desired conditions for the “Synchronize” keyword.
Statistics show 0 items deleted for relationships	Correct behavior. ServiceNow automatically deletes the existent relationships for Cis deleted during the CI Type population process
Lack of permissions for Manager role	Ask your ServiceNow administrator to include the x_nexsa_cmdb_pop.manager role in the proper ACLs related to the views with permissions issues.

Time out errors

ECCResponse TimeoutException	Is the MID Server used for the given Nexthink Engine is up and running? If you are retrieving a huge amount of data reduce the value of the x_nexsa_cmdb_pop.devices_per_batch property
HTTP 0 error	Make sure that the Nexthink Engine is up and running and the endpoint is properly set for the desired address

Mid Server errors

Import process never starts	Verify that the MID Server is properly configured and verified in ServiceNow environment. If a MID Server is not needed, please verify that the Engines configured in ServiceNow are up and running.
java.lang. NullPointerException	Make sure to define a proper MID Server to reach the Nexthink Engine and check that it is up and running. Check that the Engine is up and running.
Unexpected character ‘n’ (code 110) in prolog	If using a MID Server, create a mid.eccq.max_payload_size MID Server property, with a value for the maximum payload to receive from the Engine. If there is no MID Server, set the value (in MB) of glide.rest.scripted.max_inbound_content_length_mb.

SSL Errors

Error SSL Handshake	Set to true com.glide.communications.trustmanager_trust_all
Error SSL Peer Unverified	Set to false com.glide.communications.httpclient.verify_hostname

IRE Errors

Cis not inserted

A) Ensure the discovery source sys_choice has been created. As explained in section Identification and Reconciliation Engine (IRE) of the CMDB installation and configuration guide B) Ensure that an identification rule and identification entries are created for the target table as explained on section Identification and Reconciliation Engine (IRE) of the installation and configuration guide.

General Query-Related Errors

The following errors can be related to invalid queries sent to the Nexthink Engine. There are several options for the query to be wrong.

Invalid username/password combo (HTTP 401/403)

If the population failed and the log shows a HTTP 401 or 403 status error message related to wrong user and password, it means that the authentication profile used to connect to the Nexthink Engine is not properly configured. The error message looks like:

Method failed: (/2/query/) with code: 401 - Invalid username/password combo

In such case, please edit the Authentication Profiles used for the Engine and set the proper user and password.

Hint: the user and password to be used are those used to connect to the Nexthink Web API V2.0, for instance, via the NXQL Editor.

Nexthink Categories missing (HTTP 400)

If the population failed and the log shows a HTTP 400 status error message related to a query erroneously composed, it usually means that the ServiceNow CMDB categories used to provide more flexibility to the integration have not been properly loaded into the Nexthink Finder. The error message looks like:

Error from Nexthink API. HTTP status code '400' with query 'null'. Message: null; Error Message: Method failed: (/2/query/) with code: 400

In such case, please make sure the categories have been correctly imported into the Finder.

Field <field_name>'' of table 'device' is not defined for platform '<platform>'

If the population failed and the log shows a HTTP 400 status error message related to a given field not existing for a given platform, it is because some fields are not defined for all platforms in the NXQL Data Model. No matter if no devices of the target platform are to be retrieved or not, the NXQL will fail if the platform is selected.

To overcome this, either limit the fields to be imported to those existing for all desired platforms or select only the appropriate platforms by checking/unchecking the corresponding ServiceNow properties. However, if both platforms are needed, then you should stick to the more restrictive data model, i.e., those fields existing for both platforms. This is a limitation of the NXQL API and there is no workaround/solution for this.

No data is imported from Engine

Please, note that this resolution tip refers to the import process performed in legacy mode (IRE module is disabled).

If the “can insert” or “can update” columns are set as true in the CI Types table and no data is imported or updated, a lot of potential issues could be the cause of it. To narrow down the root cause we recommend to follow the below steps.

1. Make sure the Finder Categories have been properly modified to select desired conditions for the “Synchronize” keyword.

2. If none of the above helped, it is recommended to check the data sources and the import set data:

• Ensure the NexthinkData.xml attachments file is correctly populated on the data sources:

Go to “Nexthink CMDB Connector > Data Sources” module and click on the data source for the CI Type affected

Download the attachment and confirm that expected data is included on the file:

• Check if the import set tables are populated

Please, find below a reference to find the import set tables for each CI type

CI Type	Import Set table name
Workstation	x_nexsa_cmdb_pop_import_set_workstation
WindowsServer	x_nexsa_cmdb_pop_import_set_windowsserver
User	x_nexsa_cmdb_pop_import_set_user
Software	x_nexsa_cmdb_pop_import_set_software
Service	x_nexsa_cmdb_pop_import_set_service

And, for each relation:

CI Relation	Import Set table name
User-Service	x_nexsa_cmdb_pop_import_set_user_service
User-WindowsServer	x_nexsa_cmdb_pop_import_set_user_windowsserver
User-Workstation	x_nexsa_cmdb_pop_import_set_user_workstation
WindowsServer-Service	x_nexsa_cmdb_pop_import_set_windowsserver_service
WindowsServer-Software	x_nexsa_cmdb_pop_import_set_windowsserver_software
Workstation-Service	x_nexsa_cmdb_pop_import_set_workstation_service
Workstation-Software	x_nexsa_cmdb_pop_import_set_workstation_software

If you can find in both data source and Import set table that the information is coming as expected, we can discard the Nexthink Engine as the root cause.

Therefore, in this case, it is recommended to focus on investigating issues with ServiceNow target tables (Ensure the sys choice property is active, double-check ACL permissions and application access for the CMDB table, Investigate potential transformation errors, etc)Statistics show 0 items deleted for relationships

For some relationship population statistics, the Items deleted column will show the 0 value, instead of the expected number of relationships deleted. This is due to how ServiceNow works: if one of the CIs of a certain relationship is deleted during the CI Type population process, ServiceNow automatically deletes the existent relationships for such element. Therefore, when the CI Relationships population runs after the CI Types population finishes, expected relationships have already been deleted.

Lack of permissions for Manager role

By default, when Nexthink CMDB application is installed, the x_nexsa_cmdb_pop.manager role could have a lack of permissions for some of the application modules due to existing ACLs. This is because part of the information shown in the Nexthink CMDB modules belongs to some ServiceNow tables with permission restrictions.

Please see below samples for some of the screenshots related to this issue.

Security Constraints prevent access to requested page

Part of the query on sysauto_script has been ignored.

Blank information for some columns.

To solve this situation, please ask your ServiceNow administrator to include the x_nexsa_cmdb_pop.manager role in the proper ACLs related to the views with permissions issues.

Please see in the following Table the list of some of the ACLs to be updated:

ACL name
sys_properties
sys_properties.name
sysauto_script
sys_metadata.sys_scope
ecc_agent
ecc_agent.*
ecc_agent_property
scheduled_import_set
sys_data_source
sys_db_object
sys_db_object.*
cmdb_rel_type
sys_auth_profile

Timeout Errors

The following are some of the timeout-related errors that can occur.

ECCResponseTimeoutException

Sometimes the log shows an ECC timeout error. This happens because the ServiceNow instance waited too much and is not able to retrieve the information before the timeout expires. The error message looks like this:

Code

com.glide.ecc.ECCResponseTimeoutException: No response for ECC message request with 
sysid=<sys_id> after waiting for 300 seconds in ECC Queue

This error can appear due to different reasons:

• The MID Server is down.

• The XML REST response from the Nexthink Engine is too big and the MID Server consumes so much time to retrieve it.

To fix this, please make sure that the MID Server used for the given Nexthink Engine is up and running.

If the error is caused by the retrieval of a huge amount of data (especially during the importation of CI Relationships), a possible solution could be to reduce the value of the x_nexsa_cmdb_pop.devices_per_batch property, thus reducing the number of devices retrieved per batch.

HTTP 0 error

Logs can also state an HTTP status code ‘0’ error message. This error arises when the Nexthink Engine is not reachable for some reason, such as the Engine being shut down, the endpoint being erroneously configured, the DNS not being able to resolve the address, etc. The error message could look like any of the followings:

Code

Error from Nexthink API. HTTP status code '0' with query 'null'. Message: null; 
Error Message: The request failed: java.net.SocketTimeoutException: 
connect timed out when posting to <endpoint_address>

Code

Error from Nexthink API. HTTP status code '0' with query 'null'. Message: null; 
Error Message: The request failed: Unknown host

To fix this, please make sure that the Nexthink Engine is up and running and the endpoint is properly set for the desired address.

Mid Server Errors

Import process never starts

If the SI Nexthink or SI Nexthink Relationship scheduled scripts were launched and the status fields of the records do not change from IDLE to PROCESSING for a while, it means that the import process is not happening.

In such a case, please verify that the MID Server is properly configured and verified in ServiceNow environment. If a MID Server is not needed, please verify that the Engines configured in ServiceNow are up and running.

java.lang.NullPointerException

Sometimes, instead of showing the HTTP status code ‘0’ error message, the log shows the java.lang.NullPointerException. This error seems to be caused when the MID Server is not defined in the configuration for a given Nexthink Engine and such Engine is not accessible. The error shown is:

Code

java.lang.NullPointerException

To fix this, please make sure to define a proper MID Server to reach the Nexthink Engine and check that it is up and running. Please also check that the Engine is up and running.

Unexpected character ‘n’ (code 110) in prolog

Sometimes, when retrieving data from an Engine, the next exception may arise:

Code

com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character 'n' (code 110) 
in prolog; expected '<’

This message might mean that the ServiceNow instance exceeded the maximum payload size to receive. Consequently, the response might be broken and some additional configuration could be necessary in order to fix this and be able to receive a bigger amount of data from the Engine.

In the case of using a MID Server, it is recommended to create a mid.eccq.max_payload_size MID Server property for the given MID Server, with a value for the maximum payload to receive (in bytes) from the Engine. If there is no MID Server in place to connect to the Engine, the _glide.rest.scripted.max_inbound_content_length_mb s_ystem property may be conveniently modified to set that value (in MB).

MID Server memory issues

It has been observed in appliances with a large number of CIs, that some import processes with a big input load (I.E: Workstation-Software relationship) are not able to complete. If the logs report some ECC timeout issues and/or the error java.lang.outofmemoryerror java heap space, most likely this happened due to the mid server required more memory than the currently allocated and thus, the MID Server crashed.

To solve this incident, it is recommended to increase the heap space allocation in the agent/conf/wrapper-override.conf file of the MID Server:

Mid server override.conf file

The value that must be set on the property wrapper.java.maxmemory depends on the Server capacity and the amount of data to be inserted. This property is commented by default, therefore, it is necessary to uncomment it first.

Not trusted certificates in Quebec release

In Quebec, ServiceNow is more restrictive about how the MID Server deals with certificates. That means that if the connection to the Engine is done through a MID Server, with a non-trusted certificate, the MID Server will not allow the communication, throwing the following error in the logs:

Code

NexthinkCMDBPopulator::DataRetrieval:_processResponse : The request failed: Request not sent 
to uri= https://myURI:1671/2/query/?format=xml&hr=true&platform=windows,mac_os&query=
(select%20(name%20last_seen%20platform)%20(from%20device)) : 
org.apache.commons.httpclient.HttpException: Session contains no certificates - Untrusted

To solve this problem, the recommendation is to Import a self-signed certificate to MID Server JRE cacerts. (https://docs.servicenow.com/bundle/paris-servicenow-platform/page/product/mid-server/task/add-ssl-certificates.html ).

More details in the official ServiceNow documentation:

https://docs.servicenow.com/bundle/quebec-servicenow-platform/page/product/mid-server/concept/mid-security-checks.html

https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0867397

SSL Errors

Error SSL Handshake

If the population did not happen, the values for the status column in CI Types or CI Relationships lists are set to ERROR or PROCESSING_ERROR and in the system logs appears a message like the following:

Code

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: 
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: 
unable to find valid certification path to requested target

This error happens when Nexthink CMDB Populator is not using MID Server to retrieve the data and the Nexthink Engine does not have a valid SSL certificate. In such case, please modify the system property named com.glide.communications.trustmanager_trust_all should be set to true value.

Error SSL Peer Unverified

If the population did not happen, the values for the status column in CI Types or CI Relationships lists are set to ERROR or PROCESSING_ERROR and in the system logs appears a message like the following:

Code

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

This error happens when Engines are not configured to use MID Server to retrieve the data and the Engine does not have a valid SSL certificate (e.g., self-signed). In such case, please set to false the system property named com.glide.communications.httpclient.verify_hostname. This is not recommended for production environments. Modify this value only if you are aware of its implications

IRE Errors

No data inserted in CMDB tables

If the data is imported using IRE module, it is recommended to search in logs to search for some errors:

No Choice found in the sys_choice table for the target table

This scenario triggers the below errors in logs:

Message	Source
INVALID_INPUT_DATA In payload invalid value for field [FIELD]'. You cannot have non-cmdb class name [] here.	identification_engine
Configuration Item not processed. IRE Identification Engine marked this Configuration Item as an Erroneous in the importSet under processing [PAYLOAD] "message":"In payload invalid value for field [FIELD]'. You cannot have non-cmdb class name [] here."	x_nexsa_cmdb_pop

These errors are caused by the value incoming in the payload that is attempted to be inserted in the field displayed in the log error. In summary, this value is a sys_choice and it was not possible to be inserted in the target table because there is no choice record created for the CI table with the value inserted by the payload.

To fix this it is recommended to create the required choice in the sys_choice table. It can be accomplished manually or it can also be an automatic process if the property “Select if it is possible to insert values into sys_choice table for choice list fields” in Nexthink CMDB Connector > Properties is activated and the scheduled job executed once again.

This can be potentially caused by the issues explained in sections Discovery_source choice not created and No Windows Servers are inserted of this guide.

Identification rules not created.

This scenario triggers the below errors in logs:

Message	Source
IDENTIFICATION_RULE_MISSING Identity Rule Missing for table [TABLE]	identification_engine
INVALID_INPUT_DATA In payload invalid value for field '[FIELD]'. You cannot have non-cmdb class name [] here.	identification_engine com.glide.ui.ServletErrorListener

To overcome this situation, it is necessary to ensure that an identification rule and identification entries are created for the target table as explained in section Identification and Reconciliation Engine (IRE) of the installation and configuration guide.

Discovery_source choice not created

In IRE, all CIs are inserted in tables with the field “Discovery source” containing the information regarding the engine this information is coming from.

Since this is a choice field, one of the potential issues that might cause the CIs not inserted is that the choice for the Nexthink Engine might not be created. Thus, it is mandatory to ensure that there is a choice created in the sys choice table with the below details:

Table	cmdb_ci
Element	discovery_source
Language	en
Value	nexthink_cmdb_connector
Label	Nexthink CMDB Connector

If there is no choice listed, It can be created manually or it can also be an automatic process if the property “Select if it is possible to insert values into sys_choice table for choice list fields” in Nexthink CMDB Connector > Properties is activated and the scheduled job executed once again.

No Windows Servers are inserted.

If all data from all CIs are inserted except for Windows Server, double-check that there is a choice created in sys choice with the below details:

Table	cmdb_ci_server
Element	sys_class_name
Language	en
Label	cmdb_ci_win_server
Value	cmdb_ci_win_server

If there is no choice listed, It can be created manually or it can also be an automatic process if the property “Select if it is possible to insert values into sys_choice table for choice list fields” in Nexthink CMDB Connector > Properties is activated and the scheduled job executed once again.