Troubleshooting

This document provides comprehensive information about the introduction and concepts around Nexthink Chatbot SDK, its API and use cases. The information contained herein is subject to change without notice and is not guaranteed to be error-free. If you find any errors, please report them to us via Nexthink Support Portal. This document is intended for readers with a detailed understanding of Nexthink technology.

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

Installation checklist

The following are the steps needed in order to validate that Chatbot SDK has been properly installed and configured, and is running without any problems:

Check that all services have started. List Docker running processes to ensure that everything is running:

$ docker ps

If any process is restarting, check the logs following the instructions in the Common errors section.

Check that the ingress port is open. By default, it is 8090.

Option 1: From the appliance itself:

$ netstat -atn | grep 8090

If this command doesn’t return anything, check if the configured port on /var/nexthink/nexthink-chatbot-adapter/.env is correct. Otherwise, change it and restart the services with:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Option 2: From another machine:

Code

>ping 192.168.5.15 
>telnet 192.168.5.15 8090

If the machine or the port is not visible from the outside, check your network and port forwarding configurations.

Validate the API request to list the topics without the API key. This can be done either from the appliance itself or from the outside with Postman or Rested browser add-on.

Code

$ curl -k https://localhost:8090/v1/topics 
Invalid API key

If the result is different, check that the previous steps were done correctly.

Validate the API request to retrieve the API key.

$ curl -k https://user:password@localhost:8090/v1/api_key

If this request returns a 404 not found error, then the API key must be set again. If it fails with a 401 unauthorized error, then review the user and password that are being used. Please note that if the user and password contain special characters, they must be URL encoded when using curl.

Validate the API request to list the topics.

$ curl -k -H "X-API-key:" https://localhost:8090/v1/topics

If this request fails with a 401 unauthorized error, then review the API key which is being used.

Check remote actions execution with Finder.

Connect Finder to Portal using the same credentials which are configured in the chatbot adapter. Trigger a remote action for a device. If triggering remote actions is not enabled, review the user profile configuration.

Validate the API request to retrieve the information of a given device.

$ curl -k -H "X-API-key:" https://localhost:8090/v1/device/{device_name}

Refer to the Common errors section if you encounter an error.

Test the API request to execute a remote action.

$ curl -k -H "X-API-key:" -X POST https://localhost:8090/v1/remote_action/{ra_id}/{device_uid}

Refer to the Common errors section if you encounter an error.

Check the API request to see the result of a remote action execution.

$ curl -k -H "X-API-key:" https://localhost:8090/v1/remote_action/{ra_id}/{device_uid}

Refer to the Common errors section if you encounter an error.

Check the API request to retrieve a topic.

$ curl -k -H "X-API-key:" https://localhost:8090/v1/topic/{topic_id}/device/{device_uid}/{platform}

Refer to the Common errors section if you encounter an error.

Common errors

Remote actions are not executed

Problem: Remote actions are not being executed and there is a log entry that reads:

[RemoteActionManagerDebug] radispatcher.go:56: Remote action not configured for manual execution

Solution: Configure the remote action in Finder for manual triggering and save it.

Remote actions do not have an investigation

Problem: After installing Chatbot SDK, no investigation can be found in the remote actions that require it.

Solution: Create an investigation that retrieves all the laptops and desktops that run under Windows and assign it to all the required remote actions by dragging it into an appropriate place.

Permission denied when starting the services after the installation

Problem: The following message is printed after Chatbot SDK is installed:

[...] Got permission denied while trying to connect to the Docker daemon socket at unix [...]

Solution: Reboot the computer. The installation process assigns all the necessary permissions to the user but sometimes it is necessary to perform a reboot to use them effectively.

Configuration changes not applied

Problem: Changes do not show in the configurations (environment variables or topics configuration).

Solution: When changing configurations, ensure that one of these procedures is followed:

Environment and installation changes (.env file changes)

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Topics, Engines list or remote action changes (YAML files changes)

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$./scripts/nexthink-chatbot-adapter-create-cached-fields.sh [Only if topics changes] 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Invalid credentials

Problem: Chatbot is receiving a 401 error code and the following log error is printed:

[Discovery][nxqlclient.go:65]: [401] Your credentials are invalid.

Solution: Check that the Nexthink user and password are correct in the .env file and update if needed. Apply the changes via:

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

The API key needs to be regenerated in order to change the credentials and it has to be shared with the chatbot. Refer to the Authentication documentation.

Invalid default port 8090

Problem: While trying to start the services, the user receives the following response from Docker Compose:

[...] Bind for 0.0.0.0:8090 failed: port is already allocated

Solution:

Change the CHATBOT_ADAPTER_PORT variable in /var/nexthink/nexthink-chatbotadapter/.env
Load the Docker images again.

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Invalid subnet or connectivity lost

Problem: While trying to start the services, Docker Compose chooses a subnet that potentially collides with the one used by the host.

Solution: Stop the containers.

Code

$ cd /var/nexthink/nexthink-chatbot-adapter 
$ docker compose down

Include the networks section in the docker-compose.yaml file just after version and at the same level as services. Be sure to select a suitable subnet.

Code

version: '3.2' 
  networks:      
 default:  
       driver: bridge       
       ipam:    
          config:  
            - subnet: 10.10.100.0/24 
  services 
  ...

Load the Docker images again:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Authentication failed in the HTTP client (browser, Postman…)

Problem: When using a browser or another HTTP client, the client complains that it cannot authenticate or cannot get any answer.

First, ensure that you have already checked that:

The API key is valid.
The requests work using curl from a terminal in the local machine or from another machine. For example, to retrieve the list of topics:

$ curl -k -H "X-API-key:" https://{hostname}:8090/v1/topics

This can happen because your HTTP client is not allowed to connect to Chatbot SDK. The SDK uses a self-signed SSL certificate (not trusted).

Solution: To connect correctly to Chatbot SDK from an HTTP client, use one of the following options:

Option 1: Change the self-signed certificate to a trusted SSL certificate by following the Custom certificate installation guide.
Option 2: You can change the SSL verification parameters in the HTTP client.
- For example, in Postman this can be fixed by turning off SSL certificate verification in the General settings.
- Refer to the documentation of your specific HTTP client to change the SSL certificate validation options.

Service is not working after the installation of a custom certificate

Problem: Service is not responding to any HTTP request, chatbotingress service is restarting continually and logs show an error message:

open /certs/ca_certificates.crt: permission denied

Solution:

Check certificate file names, ca_certificates.crt for the certificate file and key.pem for the key. Ensure proper file permissions: 755 for the directory and 644 for the files, as described in the Custom certificate installation guide.
Check if the format of the certificate and the key are X.509. You may convert your key to the proper format using the following commands:

openssl rsa -in .key -out key.pem -outform PEM

- Restart the docker services

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

REST API: 500 error code

Problem: Chatbot SDK is returning a 500 error code and the logs show a critical error:

configreader_1 ... [CRI][ConfigReader][configreader.go:121]: Config value already exists for [topic_name]

Also, when checking the status of the services, the configreader service reports the status Restarting (1), as follows:

Code

$ docker ps 

configreader "./configreader -log…"   About a minute ago   Restarting (1)
10 seconds ago    nexthink-chatbot-adapter_configreader_1

The error is generated because there are two topics with the same name.

Solution:

Review the topic name inside the files in the /var/nexthink/nexthink-chatbotadapter/config/topics directory. [topic_name] described in the log above is repeated in another file.
Restart the services:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

REST API: 502 error code

Problem: The chatbot is receiving a 502 error code and one of the following messages:

Invalid Server Error
The Engine returned an error while processing the Remote Action - <error_information>
All Engines failed
Another type of error

These errors point out a problem after a remote action is dispatched to Portal. In order to fully understand them, review the log as the causes can vary.

Solution:

It is recommended to always configure a databaseUID in the engines.yaml file. This way, the remote actions dispatch targets a specific Engine and it is easier to narrow down the errors.

The following are general descriptions for each of the possible error outputs:

Invalid Server Error: Portal always answers with an error code that was caused by a lack of proper permission, or a failure of another kind.
Engine returned an error while processing the RA - : Portal returned a success code, but the remote action dispatch failed. <error_information>refers to this failure.
All Engines failed: All Engines to which the remote action was dispatched returned a problem. To avoid this, configure databaseUID in the engines.yaml file.
Another kind of error information: Portal returned an error, but the Chatbot SDK API did not identify it as an Invalid Server Error. Please read the logs and the output to obtain more information.

REST API: 401 error code

Problem: The chatbot is receiving a 401 error code with the message Invalid API Key

Solution:

Check that the API Key is correctly added in an extended header X-API-key in the HTTP requests sent by the chatbot. Use the following command to retrieve the current API Key:

$ curl -k https://:@localhost:8090/v1/api_key

Nexthink_User is the Portal user configured in /var/nexthink/nexthink_chatbot_adapter/.env

If required, generate a new API Key and share it with the chatbot as explained in the Authentication guide.

REST API: 404 error code

Problem: The chatbot is receiving a 404 error code with the message Device or Remote Action not found when requesting the status of a remote action or when executing synchronously.

This problem may be originating from the underlying NXQL query requesting the status fails due to the device not being in the local Chatbot SDK cache or due to the remote action not being found.

Solution: Check the name of the remote action in remote_actions.yaml. The name is case-sensitive and should match the name of the remote action in Finder.

REST API: 404 error code

Problem: The chatbot is receiving a 404 error code with the message Device not found when requesting data from an existing device.

This problem may be caused by the device not being in the local Chatbot SDK cache. There could be several reasons for this behavior:

The device has not been detected for an extended period of time by Engine.
The device is not generating user activity in the way it was expected or the discovery process has failed for some reason.

Solution:

Validate whether the discovery process runs successfully for all Engines by checking the logs of the discovery service:

Code

$ cd /var/nexthink/nexthink-chatbot-adapter 
$ docker compose logs discovery

Check for errors such as:

[engine_10-10-10-1_1672] Discovery failed for this Engine. Information has not been updated due to previous error.

The Web API port may not be the proper one. Verify that all the information in the engines.yaml file is correct and fix it accordingly. Refer to the Configuration changes not applied section on how to restart the services properly after fixes have been applied.

Check the query configured in the discovery.yaml file. There are two queries, one of them is commented out.
Execute the active NXQL query in the NXQL editor on one of the Engines to check if:
- the device exists in Engine.
- the device has sent data to Engine, showing existing rows for that device in the user_activity or executions objects, depending on the active query, in the last USER_ACTIVITY_PERIOD days.

Replace <USER_ACTIVITY_PERIOD> with your configured period in the NXQL Editor. The default value is 30 days.

If the device does not appear in the query, then it is not included in the Chatbot SDK cached database. It was not reporting activity and therefore it cannot be found.

Check the execution of the commented query in the NXQL editor. If it returns information about the device then change the configured commented query and restart Chatbot SDK services:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

REST API: 429 error code

Problem: The chatbot is receiving a 429 error code with the message Throttle limit reached. This is caused by Chatbot SDK reaching the max limit of transactions per second to access either Portal, Engines or the internal cache.

Solution:

When receiving this error, the chatbot will ask the user to try the operation later after a few minutes. The limit on the number of transactions can be configured using parameters in /var/nexthink/nexthink_chatbot_adapter/.env
A less recommended solution, which is valid only for very specific cases in environments with low load on the Engines or Portal, is to increase those limits by no more than 10% in /var/nexthink/nexthink_chatbot_adapter/.env

Code

MAX_CACHE_REQUESTS_PER_SECOND=500 
MAX_PORTAL_REQUESTS_PER_SECOND=20 
MAX_ENGINE_REQUESTS_PER_SECOND=40

This is not recommended for Engines or Portal under a heavy load.

REST API: Ingress is returning a Client.Timeout error

Problem: Chatbot SDK returns the following error:

Code

chatbotingress_1       | 2019-12-31T05:10:54-05:00 [DEB][Ingress][router.go:32]: Post 
http://remoteactionmanager:8003/v1/remote_action/<remote_action_name>/<remote_action_id>: 
net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Solution: Check the value which has been assigned to the parameter NXQL_CLIENT_TOPIC_TIMEOUT in the .env file. Generally, it is not necessary to change its default value, however, in this case you might need to set it to a higher value.

REST API: Topic diagnosis gives back a set of null fields

Problem: The JSON retrieved from a topic contains a list of fields that are all null, for example:

Code

"fields": { 
 "action:Get Outlook Plugin Crash Details/CrashedPluginList": null, 
 "action:Get Outlook Plugin Crash Details/Execution status": null, 
 "action:Get Outlook Plugin Crash Details/Execution status details": null, 
 "action:Get Outlook Plugin Crash Details/Last execution time": null, 
 "action:Get Outlook Plugin Crash Details/Last target time": null, 
 "action:Get Outlook Plugin Crash Details/LastCrashedPluginDate": null, 
 "action:Get Outlook Plugin Crash Details/LastCrashedPluginName": null, 
 "action:Get Outlook Plugin Crash Details/MaximumCrashCountPluginName": null, 
 "action:Get Outlook Plugin Crash Details/Number of days since last execution": null,         
 "action:Get Outlook Plugin Crash Details/Target status": null 
    }

This issue is also printed in the Chatbot SDK log:

[DEB][TopicManager][topicmanager.go:395]: The value for the field action:Get Outlook Plugin Crash Details/CrashedPluginList is NULL. No comparison will be performed.

Solution:

Check with your Nexthink operator if the remote action shown in the example above Get Outlook Plugin Crash Details is scheduled to be automatically launched and has been executed, as described in the Installation and configuration guide.

REST API: Error “NXQL query returned no results” in diagnosis error

Problem: The JSON retrieved from a topic contains a diagnosis error with the message “NXQL query returned no results”. For example:

Code

{ 
 "diagnosis_error": { 
 "checked_issue": "(select name (from device (with execution_error 
                          (where device (eq device_uid (md5 %1))) 
                          (where application (eq name (pattern \"*outlook*\"))) 
                          (compute number_of_application_crashes number_of_application_not_responding_events) 
                          (between now-7d now))))", 
 "errors": [ 
              { 
 "code": 2, 
 "message": "Error executing NXQL query: [404]The NXQL query returned no results. 
                  Query: (select name (from device (with execution_error 
                          (where device (eq device_uid (md5 %1))) 
                          (where application (eq name (pattern \"*outlook*\"))) 
                          (compute number_of_application_crashes number_of_application_not_responding_events) 
                          (between now-7d now))))" 
              } 
        ] 
    } 
}

The issue is also printed in the Chatbot SDK log:

Code

[ERR][TopicManager][topicmanager.go:332]: Error executing NXQL query: 
[404]The NXQL query returned no results. Query: (select name (from device 
(with execution_error (where device (eq device_uid (md5 %!)(MISSING))) 
(where application (eq name (pattern "*outlook*"))) 
(compute number_of_application_crashes number_of_application_not_responding_events) 
(between now-7d now))))

Solution:

Check if the device has had any recent user activity using Nexthink Finder. If not, it is normal not to receive any information about the device.
Validate that the NXQL query syntax is correct.
- Check the where clause in case there is a non-existing condition.
- Check if there are any other typographical errors.

REST API: Error: “Remote Action '<remote_action_id>' not configured properly”

Problem: The JSON retrieved from a topic contains a diagnosis_error and the message: Error: Remote Action ... not configured, for example:

Code

[{"code":7,"message":"Error while preparing diagnosis. Error: Remote Action 
'<remote_action_id>' not configured properly. Please check that it exists in 
your configuration files."}]

This message indicates that Chatbot SDK is trying a diagnosis using a remote action that does not exist in /var/nexthink/nexthink-chatbot-adapter/config/topics/remote_actions.yaml. To fix the problem, add it to the file and restart the Docker services.

If the topic contains only one diagnosis, Chatbot SDK will return a 500 status code.
If by any chance other diagnoses are evaluated and the result is successful, a 200 status code along with the usual response message structure will be generated.

REST API: Chatbot SDK is not retrieving all fields of a remote action

Problem: Remote actions fields are retrieved even though they are included in the configuration files.

Solution: Check that you are always using the latest version of all remote actions.

Table of Error Codes

Chatbot SDK logs an error message with a specific code. The following are the error codes that you may encounter:

Code	Type
0	Remote Action Client error
1	Remote Action Execution error
2	NXQL Client error
3	NXQL Query format error
4	Remediation Condition error
5	NXQL Fields error
6	Remediation Comparison error
7	Remote Action Configuration error

Logs Handling

How to read logs

Logs are written in the standard output of each microservice. You can retrieve all of them as follows:

Code

$ cd /var/nexthink/nexthink-chatbot-adapter 
$ docker compose logs

How to increase the log level

The log level is set in Chatbot SDK .env file with the variable LOG_LEVEL

$ vi /var/nexthink/nexthink-chatbot-adapter/.env

The default value is 4 (Info). The valid values are:

1 (Critical)
2 (Error)
3 (Warning)
4 (Info)
5 (Debug)
6 (Trace)

To take the new value, you need to restart the affected microservices by typing:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Recovery procedures

Restart the Docker containers

To list the status of the Docker containers:

$ docker ps --all

If any containers are down, restart them. To restart the Docker containers:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Refreshing the local cache manually, the discovery process

The cache is refreshed automatically with a frequency defined in the file /var/nexthink/nexthinkchatbot-adapter/.env and with the variable CHATBOT_CACHE_REFRESH

There are cases where it might be required to force the cache refresh manually, for example, when the waiting time to refresh the cache automatically is too long as indicated by some tests. This can be caused by:

New Engines being added into the system, in /var/nexthink/nexthink-chatbotadapter/config/engines.yaml
Execution of new remote actions
Addition of new devices

This requires restarting the discovery microservice as follows:

Code

$ cd /var/nexthink/nexthink-chatbot-adapter 
$ docker compose restart discovery

Updating the list of fields in the local cache

You may need to update fields in the cache in order to gain performance or to fix certain configuration issues. The information stored in the cache is listed in the /var/nexthink/nexthink-chatbotadapter/config/cached_fields.yaml

This file is automatically generated during the installation and upgrade process based on the NXQL queries found in the selected topic files.

Updating the cached fields automatically

The fields are updated automatically by default every time there is an installation or upgrade. The content of the cached fields is created in the config/cached_fields.yaml file programmatically.

You have to manually launch the script when changes to the configuration of the topics are made, the topic files are modified or new topics are added.

To regenerate the cache fields follow these steps:

Start SSH-client, connect to VM and login with the Nexthink user.
Go to the directory where the installation zip was extracted, usually in the user’s home directory.
Add execution permission to the script, if needed:

$ chmod +x scripts/nexthink-chatbot-adapter-create-cached-fields.sh

Execute the script:

$ ./scripts/nexthink-chatbot-adapter-create-cached-fields.sh

Restart the running containers in order to apply the changes:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Edit the cache fields manually

The list of fields to be cached can be customized. Since the config/cached_fields.yaml file is automatically created, it is recommended to perform the following operation only for troubleshooting reasons:

Start SSH-client, connect to VM and login with the Nexthink user.
Go to Chatbot SDK installation directory:

$ cd /var/nexthink/nexthink_chatbot_adapter/config

Open and edit the cached fields file:

$ vi cached_fields.yaml

Each line represents a cache field that is stored for every device during the discovery process.

Restart the running containers in order to apply the changes:

Code

$ cd /home/nexthink/nexthink-chatbot-adapter-1.1.0 
$ ./scripts/nexthink-chatbot-adapter-restart.sh

Rollback

While executing an update, it is possible that the process is unable to finish successfully. If this is the case, Chatbot SDK will perform a rollback by itself. Despite the error messages being shown in the logs and command line, everything will be restored to its previous state automatically.

What the rollback reverts, depends on the updating stage in which the error occurred. Refer to the Updating to newer version documentation.

Support script

In order to retrieve useful information for troubleshooting purposes, a support script is provided along with the installation.

Launching the support script

Go to the directory where the installation zip was extracted, normally in the user’s home directory.
Add execution permissions to the script in case they are needed:

$ chmod +x scripts/nexthink-chatbot-adapter-support-script.sh

Execute the script. In order to retrieve all the necessary information, Chatbot SDK should be up and running. If it is not running, important information about the status of the containers will not be retrieved.

$ ./scripts/nexthink-chatbot-adapter-support-script.sh

The script returns to the console location of the compressed file with the results, which will be in the /tmp directory, with the name nexthink-chatbot-adapter-support, and suffixed with a timestamp. For example, for some information generated on March 4th, 2020, at 15:16, the path would be /tmp/nexthink-chatbot-adapter-support-2020-03-04-15-16-47.tar.gz

Content of the output file

The compressed output file consists of the following elements:

config (folder): copy of the original user configuration directory which contains:
- topics (folder): a folder containing topics and remote actions .yaml files
- .yaml configuration files: cached_fields.yaml, discovery.yaml and engines.yaml
docker_inspect (folder): contains a .txt file per container with the docker_inspect information, for example, docker-inspect-nexthink-chatbot-adapter_chatbotingress_1.txt
docker_logs (folder): contains the docker logs files, both, those created by the restart script and a log file created by the support script. The file content is sorted by ascending timestamp.
.env: copy of the .env configuration file without any password
general_information.txt: text file which contains the following:
- The number of installed release versions
- Permissions for the /config directory and /topics contained within it
- Docker image information: REPOSITORY, TAG, IMAGE ID, CREATED and SIZE
- Docker container information: CONTAINER ID, IMAGE, COMMAND, CREATED, STATUS, PORTS and NAMES
- Docker stats: CONTAINER ID, NAME, CPU %, MEM USAGE/LIMIT, MEM %, NET I/O, BLOCK I/O and PIDS
Folders with the result of the install, update, restart and uninstall processes

Last updated 2 months ago