Troubleshooting guide

Introduction

This document provides comprehensive information on Troubleshooting for the Nexthink Event Connector. This document provides a detailed description of procedures that can help in analyzing problems, as well as guidance to debug and solve them.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us via Nexthink support portal.

This document is intended for readers with a detailed understanding of Nexthink technology and Splunk technology, as well as some understanding of concepts such as REST messages, HTTP errors, and some basic security terms.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

Nexthink Event Connector – Troubleshooting Guide

Version: 1.5.0

Last Revision: 16/10/2023

Overview

The log file where the following error conditions may be detected is located in the machine hosting the Nexthink Event Connector service, by default at /var/log/nxeventconnector/nxeventconnector.log. This default configuration can be conveniently modified, see the Initial configuration section of the Installation and configuration guide document.

Dependencies

No Internet connection

If the machine where the Nexthink Event Connector is installed does not have an Internet connection, be aware that the following python dependencies must be satisfied:

python-setuptools

pytz==2021.1

requests==2.25.1

azure-core==1.12.0

azure-identity==1.5.0

azure-storage-blob==12.8.0

azure-storage-file-datalake==12.3.0

requests_oauthlib==1.3.0

cachetools==4.2.4

It is necessary to manually install RPMS with such packages in the machine where the Nexthink Event Connector will be set up.

Default configuration files not modified

The following errors may occur if the default configuration files have not been modified.

No connection adapters were found

The service has started but never reaches the Engine and the log file displays something like:

CODE

ERROR orchestrator [-] No connection adapters were found for
'<protocol>://<your_splunk_instance>:<port>/services/collector' for
<protocol>://<your_splunk_instance>:<port>/services/collector

In this case, please edit the ENGINES section in the /etc/nxeventconnector/config.conf file to ensure that the endpoint for each Engine has the correct values. It is necessary to also edit the URI parameter of the specific integration section (either SPLUNK_HEC or SERVICENOW) to ensure it has the expected value.

Query is not a String

The service has started, but it never reaches the Engine and the log file displays an error similar to the one below:

ERROR orchestrator [-] Query is not a string

To fix this issue, please check in /etc/nxeventconnector/events.conf to verify that all the queries are filled and correct.

Connection errors

The following are some of the connection related errors that can occur.

HTTP 401: Token is required (Splunk)

The error message shows that the nxeventconnector service has started, but the data never reaches the configured Splunk instance and the log file displays an error code similar to the following:

ERROR orchestrator [-] HTTP 401: Token is required for http://yoursplunkinstance/services/collector

In this case, check in the /etc/nxeventconnector/config.conf file in the SPLUNK_HEC section to be sure that the token is correct.

HTTP 401: Credentials are required (ServiceNow)

The error shows that the nxeventconnector service has started, but the data never reaches the configured ServiceNow instance and the log file displays an error code similar to the following:

ERROR orchestrator [-] HTTP 401: credentials are required, invalid authorization

In this case, please check the SERVICENOW section in the /etc/nxeventconnector/config.conf file to ensure that the login and password are correct.

Azure Data Lake Retrieve Access token error (Azure Data Lake Storage Gen2)

The error shows that the nxeventconnector service has started, but it is unable to authenticate against Azure, the log file displays an error code similar to the following:

msal_client [-] Unable to retrieve access token, error 'unauthorized_client'

In this case, check the AZURE DATALAKE_STORAGE_GEN2 section in the /etc/nxeventconnector/config.conf file to ensure that the tenant_id, filesystem, client_id and client_secret are correct and have the correct permissions to write in the selected filesystem.

Status code 400

The service has started, but it never reaches the configured Engine and the log file displays an error similar to the following:

ERROR orchestrator [-] Status code 401 for https://...

This could be solved by updating the ENGINES section in the /etc/nxeventconnector/config.conf file to ensure that the user and password are correct for each engine.

Event connection returned exception sending

The service has started and reaches the Engine, but the data is never retrieved into the target instance. For this issue, DEBUG mode should be set. The log file may display an error message similar to the following:

DEBUG orchestrator [-] Event connection returned exception sending <JSON-DATA>

Again, please check in the SPLUNK_HEC, SERVICENOW or AZURE_DATALAKE_STORAGE_GEN2 section of the /etc/nxeventconnector/config.conf file to be sure that the endpoint for the target instance has the correct values.

Error retrieving data: HTTPSConnectionPool

The service has started, but it never reaches the Engine and the log file displays an error message similar to the following:

ERROR orchestrator [-] Error retrieving data: HTTPSConnectionPool <NXQL-query>

Please check in the ENGINES section of the /etc/nxeventconnector/config.conf file to be sure that the endpoint for each Engine has the correct values.

HTTPSConnectionPool

The service has started, but it never reaches the Engine and the log file displays an error message similar to the following::

ERROR orchestrator [-] HTTPSConnectionPool(host='...', port=...): Max retries exceeded with url: /2/query?platform=windows.....

Please check in the ENGINES section of the /etc/nxeventconnector/config.conf file to be sure that the endpoint for each Engine has the correct values.

SSL: Certificate verify failed

There are some conditions under which this error might arise:

verify_cert_engine or verify_cert_target parameters are set to true, see the Initial configuration section of the Installation and configuration guide document.
The devices with the verify_cert_* set to true are using the Python version included in the package python-2.7.5-58.el7 .
The certificates used for the devices with the verify_cert_* set to true are invalid.

In such a case, a message similar to the following may be found in the log:

ERROR orchestrator [-] Error retrieving data: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) for https://x.x.x.x:1671/2/query?platform=windows&platform=mac_os&format=json&query=(se lect%20((device%20(name))......

The best fix is to set the verify_cert_* parameters to false and restart the service.

Service not restarting

Service not starting and no error messages in the log file

In this case, the service start command is executed, but immediately the service stops in failed status. A log file exists, but no relevant errors appear inside.

Please check the user and group of the log file /var/log/nxeventconnector/nxeventconnector.log. If the user or group are different from the nxeventconnector value, please change it to the following:

sudo chown nxeventconnector:nxeventconnector /var/log/nxeventconnector/nxeventconnector.log

Restart the service. Now it should be up and running.

This issue may occur when the tools provided to check engine status, check query validation or check timezones, are executed after the first installation and just before the first service starts. These tools use the same log file as the service, but with a different user. So, if the log file did not exist when some of the tools were executed, these tools will create a log file (because the installation does not create it) with a different user/group than the nxeventconnector value.

Support

Nexthink provides support for the application in accordance with the terms and conditions of the Support and Maintenance Agreement applicable in between the customer and Nexthink. If you have any questions, please contact us via the Nexthink support portal.