Troubleshooting
Last updated
Was this helpful?
Version 6.30
Last updated
Was this helpful?
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.
The following are the steps needed in order to validate that Chatbot SDK has been properly installed and configured, and is running without any problems:
Check that all services have started. List Docker running processes to ensure that everything is running:
$ docker ps
If any process is restarting, check the logs following the instructions in the section.
Check that the ingress port is open. By default, it is 8090.
Option 1: From the appliance itself:
$ netstat -atn | grep 8090
If this command doesn’t return anything, check if the configured port on /var/nexthink/nexthink-chatbot-adapter/.env is correct. Otherwise, change it and restart the services with:
Option 2: From another machine:
If the machine or the port is not visible from the outside, check your network and port forwarding configurations.
Validate the API request to list the topics without the API key. This can be done either from the appliance itself or from the outside with Postman or Rested browser add-on.
If the result is different, check that the previous steps were done correctly.
Validate the API request to retrieve the API key.
$ curl -k https://user:password@localhost:8090/v1/api_key
If this request returns a 404 not found error, then the API key must be set again. If it fails with a 401 unauthorized error, then review the user and password that are being used. Please note that if the user and password contain special characters, they must be URL encoded when using curl.
Validate the API request to list the topics.
$ curl -k -H "X-API-key:" https://localhost:8090/v1/topics
If this request fails with a 401 unauthorized error, then review the API key which is being used.
Check remote actions execution with Finder.
Connect Finder to Portal using the same credentials which are configured in the chatbot adapter. Trigger a remote action for a device. If triggering remote actions is not enabled, review the user profile configuration.
Validate the API request to retrieve the information of a given device.
$ curl -k -H "X-API-key:" https://localhost:8090/v1/device/{device_name}
Test the API request to execute a remote action.
$ curl -k -H "X-API-key:" -X POST https://localhost:8090/v1/remote_action/{ra_id}/{device_uid}
Check the API request to see the result of a remote action execution.
$ curl -k -H "X-API-key:" https://localhost:8090/v1/remote_action/{ra_id}/{device_uid}
Check the API request to retrieve a topic.
$ curl -k -H "X-API-key:" https://localhost:8090/v1/topic/{topic_id}/device/{device_uid}/{platform}
Problem: Remote actions are not being executed and there is a log entry that reads:
[RemoteActionManagerDebug] radispatcher.go:56: Remote action not configured for manual execution
Solution: Configure the remote action in Finder for manual triggering and save it.
Problem: After installing Chatbot SDK, no investigation can be found in the remote actions that require it.
Solution: Create an investigation that retrieves all the laptops and desktops that run under Windows and assign it to all the required remote actions by dragging it into an appropriate place.
Problem: The following message is printed after Chatbot SDK is installed:
[...] Got permission denied while trying to connect to the Docker daemon socket at unix [...]
Solution: Reboot the computer. The installation process assigns all the necessary permissions to the user but sometimes it is necessary to perform a reboot to use them effectively.
Problem: Changes do not show in the configurations (environment variables or topics configuration).
Solution: When changing configurations, ensure that one of these procedures is followed:
Environment and installation changes (.env file changes)
Topics, Engines list or remote action changes (YAML files changes)
Problem: Chatbot is receiving a 401 error code and the following log error is printed:
[Discovery][nxqlclient.go:65]: [401] Your credentials are invalid.
Solution: Check that the Nexthink user and password are correct in the .env file and update if needed. Apply the changes via:
Problem: While trying to start the services, the user receives the following response from Docker Compose:
[...] Bind for 0.0.0.0:8090 failed: port is already allocated
Solution:
Change the CHATBOT_ADAPTER_PORT variable in /var/nexthink/nexthink-chatbotadapter/.env
Load the Docker images again.
Problem: While trying to start the services, Docker Compose chooses a subnet that potentially collides with the one used by the host.
Solution: Stop the containers.
Include the networks section in the docker-compose.yaml file just after version and at the same level as services. Be sure to select a suitable subnet.
Load the Docker images again:
Problem: When using a browser or another HTTP client, the client complains that it cannot authenticate or cannot get any answer.
First, ensure that you have already checked that:
The API key is valid.
The requests work using curl from a terminal in the local machine or from another machine. For example, to retrieve the list of topics:
$ curl -k -H "X-API-key:" https://{hostname}:8090/v1/topics
This can happen because your HTTP client is not allowed to connect to Chatbot SDK. The SDK uses a self-signed SSL certificate (not trusted).
Solution: To connect correctly to Chatbot SDK from an HTTP client, use one of the following options:
Option 2: You can change the SSL verification parameters in the HTTP client.
For example, in Postman this can be fixed by turning off SSL certificate verification in the General settings.
Refer to the documentation of your specific HTTP client to change the SSL certificate validation options.
Problem: Service is not responding to any HTTP request, chatbotingress service is restarting continually and logs show an error message:
open /certs/ca_certificates.crt: permission denied
Solution:
Check if the format of the certificate and the key are X.509. You may convert your key to the proper format using the following commands:
openssl rsa -in .key -out key.pem -outform PEM
- Restart the docker services
Problem: Chatbot SDK is returning a 500 error code and the logs show a critical error:
configreader_1 ... [CRI][ConfigReader][configreader.go:121]: Config value already exists for [topic_name]
Also, when checking the status of the services, the configreader service reports the status Restarting (1), as follows:
The error is generated because there are two topics with the same name.
Solution:
Review the topic name inside the files in the /var/nexthink/nexthink-chatbotadapter/config/topics directory. [topic_name] described in the log above is repeated in another file.
Restart the services:
Problem: The chatbot is receiving a 502 error code and one of the following messages:
Invalid Server Error
The Engine returned an error while processing the Remote Action - <error_information>
All Engines failed
Another type of error
These errors point out a problem after a remote action is dispatched to Portal. In order to fully understand them, review the log as the causes can vary.
Solution:
It is recommended to always configure a databaseUID in the engines.yaml file. This way, the remote actions dispatch targets a specific Engine and it is easier to narrow down the errors.
The following are general descriptions for each of the possible error outputs:
Invalid Server Error: Portal always answers with an error code that was caused by a lack of proper permission, or a failure of another kind.
Engine returned an error while processing the RA - : Portal returned a success code, but the remote action dispatch failed. <error_information>refers to this failure.
All Engines failed: All Engines to which the remote action was dispatched returned a problem. To avoid this, configure databaseUID in the engines.yaml file.
Another kind of error information: Portal returned an error, but the Chatbot SDK API did not identify it as an Invalid Server Error. Please read the logs and the output to obtain more information.
Problem: The chatbot is receiving a 401 error code with the message Invalid API Key
Solution:
Check that the API Key is correctly added in an extended header X-API-key in the HTTP requests sent by the chatbot. Use the following command to retrieve the current API Key:
$ curl -k https://:@localhost:8090/v1/api_key
Nexthink_User is the Portal user configured in /var/nexthink/nexthink_chatbot_adapter/.env
Problem: The chatbot is receiving a 404 error code with the message Device or Remote Action not found when requesting the status of a remote action or when executing synchronously.
This problem may be originating from the underlying NXQL query requesting the status fails due to the device not being in the local Chatbot SDK cache or due to the remote action not being found.
Solution: Check the name of the remote action in remote_actions.yaml. The name is case-sensitive and should match the name of the remote action in Finder.
Problem: The chatbot is receiving a 404 error code with the message Device not found when requesting data from an existing device.
This problem may be caused by the device not being in the local Chatbot SDK cache. There could be several reasons for this behavior:
The device has not been detected for an extended period of time by Engine.
The device is not generating user activity in the way it was expected or the discovery process has failed for some reason.
Solution:
Validate whether the discovery process runs successfully for all Engines by checking the logs of the discovery service:
Check for errors such as:
[engine_10-10-10-1_1672] Discovery failed for this Engine. Information has not been updated due to previous error.
Check the query configured in the discovery.yaml file. There are two queries, one of them is commented out.
Execute the active NXQL query in the NXQL editor on one of the Engines to check if:
the device exists in Engine.
the device has sent data to Engine, showing existing rows for that device in the user_activity or executions objects, depending on the active query, in the last USER_ACTIVITY_PERIOD days.
If the device does not appear in the query, then it is not included in the Chatbot SDK cached database. It was not reporting activity and therefore it cannot be found.
Check the execution of the commented query in the NXQL editor. If it returns information about the device then change the configured commented query and restart Chatbot SDK services:
Problem: The chatbot is receiving a 429 error code with the message Throttle limit reached. This is caused by Chatbot SDK reaching the max limit of transactions per second to access either Portal, Engines or the internal cache.
Solution:
When receiving this error, the chatbot will ask the user to try the operation later after a few minutes. The limit on the number of transactions can be configured using parameters in /var/nexthink/nexthink_chatbot_adapter/.env
A less recommended solution, which is valid only for very specific cases in environments with low load on the Engines or Portal, is to increase those limits by no more than 10% in /var/nexthink/nexthink_chatbot_adapter/.env
This is not recommended for Engines or Portal under a heavy load.
Problem: Chatbot SDK returns the following error:
Solution: Check the value which has been assigned to the parameter NXQL_CLIENT_TOPIC_TIMEOUT in the .env file. Generally, it is not necessary to change its default value, however, in this case you might need to set it to a higher value.
Problem: The JSON retrieved from a topic contains a list of fields that are all null, for example:
This issue is also printed in the Chatbot SDK log:
[DEB][TopicManager][topicmanager.go:395]: The value for the field action:Get Outlook Plugin Crash Details/CrashedPluginList is NULL. No comparison will be performed.
Solution:
Problem: The JSON retrieved from a topic contains a diagnosis error with the message “NXQL query returned no results”. For example:
The issue is also printed in the Chatbot SDK log:
Solution:
Check if the device has had any recent user activity using Nexthink Finder. If not, it is normal not to receive any information about the device.
Validate that the NXQL query syntax is correct.
Check the where clause in case there is a non-existing condition.
Check if there are any other typographical errors.
Problem: The JSON retrieved from a topic contains a diagnosis_error and the message: Error: Remote Action ... not configured, for example:
This message indicates that Chatbot SDK is trying a diagnosis using a remote action that does not exist in /var/nexthink/nexthink-chatbot-adapter/config/topics/remote_actions.yaml. To fix the problem, add it to the file and restart the Docker services.
If the topic contains only one diagnosis, Chatbot SDK will return a 500 status code.
If by any chance other diagnoses are evaluated and the result is successful, a 200 status code along with the usual response message structure will be generated.
Problem: Remote actions fields are retrieved even though they are included in the configuration files.
Solution: Check that you are always using the latest version of all remote actions.
Chatbot SDK logs an error message with a specific code. The following are the error codes that you may encounter:
0
Remote Action Client error
1
Remote Action Execution error
2
NXQL Client error
3
NXQL Query format error
4
Remediation Condition error
5
NXQL Fields error
6
Remediation Comparison error
7
Remote Action Configuration error
Logs are written in the standard output of each microservice. You can retrieve all of them as follows:
The log level is set in Chatbot SDK .env file with the variable LOG_LEVEL
$ vi /var/nexthink/nexthink-chatbot-adapter/.env
The default value is 4 (Info). The valid values are:
1 (Critical)
2 (Error)
3 (Warning)
4 (Info)
5 (Debug)
6 (Trace)
To take the new value, you need to restart the affected microservices by typing:
To list the status of the Docker containers:
$ docker ps --all
If any containers are down, restart them. To restart the Docker containers:
The cache is refreshed automatically with a frequency defined in the file /var/nexthink/nexthinkchatbot-adapter/.env and with the variable CHATBOT_CACHE_REFRESH
There are cases where it might be required to force the cache refresh manually, for example, when the waiting time to refresh the cache automatically is too long as indicated by some tests. This can be caused by:
New Engines being added into the system, in /var/nexthink/nexthink-chatbotadapter/config/engines.yaml
Execution of new remote actions
Addition of new devices
This requires restarting the discovery microservice as follows:
You may need to update fields in the cache in order to gain performance or to fix certain configuration issues. The information stored in the cache is listed in the /var/nexthink/nexthink-chatbotadapter/config/cached_fields.yaml
This file is automatically generated during the installation and upgrade process based on the NXQL queries found in the selected topic files.
The fields are updated automatically by default every time there is an installation or upgrade. The content of the cached fields is created in the config/cached_fields.yaml file programmatically.
You have to manually launch the script when changes to the configuration of the topics are made, the topic files are modified or new topics are added.
To regenerate the cache fields follow these steps:
Start SSH-client, connect to VM and login with the Nexthink user.
Go to the directory where the installation zip was extracted, usually in the user’s home directory.
Add execution permission to the script, if needed:
$ chmod +x scripts/nexthink-chatbot-adapter-create-cached-fields.sh
Execute the script:
$ ./scripts/nexthink-chatbot-adapter-create-cached-fields.sh
Restart the running containers in order to apply the changes:
The list of fields to be cached can be customized. Since the config/cached_fields.yaml file is automatically created, it is recommended to perform the following operation only for troubleshooting reasons:
Start SSH-client, connect to VM and login with the Nexthink user.
Go to Chatbot SDK installation directory:
$ cd /var/nexthink/nexthink_chatbot_adapter/config
Open and edit the cached fields file:
$ vi cached_fields.yaml
Each line represents a cache field that is stored for every device during the discovery process.
Restart the running containers in order to apply the changes:
While executing an update, it is possible that the process is unable to finish successfully. If this is the case, Chatbot SDK will perform a rollback by itself. Despite the error messages being shown in the logs and command line, everything will be restored to its previous state automatically.
In order to retrieve useful information for troubleshooting purposes, a support script is provided along with the installation.
Go to the directory where the installation zip was extracted, normally in the user’s home directory.
Add execution permissions to the script in case they are needed:
$ chmod +x scripts/nexthink-chatbot-adapter-support-script.sh
Execute the script. In order to retrieve all the necessary information, Chatbot SDK should be up and running. If it is not running, important information about the status of the containers will not be retrieved.
$ ./scripts/nexthink-chatbot-adapter-support-script.sh
The script returns to the console location of the compressed file with the results, which will be in the /tmp directory, with the name nexthink-chatbot-adapter-support, and suffixed with a timestamp. For example, for some information generated on March 4th, 2020, at 15:16, the path would be /tmp/nexthink-chatbot-adapter-support-2020-03-04-15-16-47.tar.gz
The compressed output file consists of the following elements:
config (folder): copy of the original user configuration directory which contains:
topics (folder): a folder containing topics and remote actions .yaml files
.yaml configuration files: cached_fields.yaml, discovery.yaml and engines.yaml
docker_inspect (folder): contains a .txt file per container with the docker_inspect information, for example, docker-inspect-nexthink-chatbot-adapter_chatbotingress_1.txt
docker_logs (folder): contains the docker logs files, both, those created by the restart script and a log file created by the support script. The file content is sorted by ascending timestamp.
.env: copy of the .env configuration file without any password
general_information.txt: text file which contains the following:
The number of installed release versions
Permissions for the /config directory and /topics contained within it
Docker image information: REPOSITORY, TAG, IMAGE ID, CREATED and SIZE
Docker container information: CONTAINER ID, IMAGE, COMMAND, CREATED, STATUS, PORTS and NAMES
Docker stats: CONTAINER ID, NAME, CPU %, MEM USAGE/LIMIT, MEM %, NET I/O, BLOCK I/O and PIDS
Folders with the result of the install, update, restart and uninstall processes
Refer to the section if you encounter an error.
Refer to the section if you encounter an error.
Refer to the section if you encounter an error.
Refer to the section if you encounter an error.
The API key needs to be regenerated in order to change the credentials and it has to be shared with the chatbot. Refer to the documentation.
Option 1: Change the self-signed certificate to a trusted SSL certificate by following the installation guide.
Check certificate file names, ca_certificates.crt for the certificate file and key.pem for the key. Ensure proper file permissions: 755 for the directory and 644 for the files, as described in the guide.
If required, generate a new API Key and share it with the chatbot as explained in the guide.
The Web API port may not be the proper one. Verify that all the information in the engines.yaml file is correct and fix it accordingly. Refer to the section on how to restart the services properly after fixes have been applied.
Check with your Nexthink operator if the remote action shown in the example above Get Outlook Plugin Crash Details is scheduled to be automatically launched and has been executed, as described in the guide.
What the rollback reverts, depends on the updating stage in which the error occurred. Refer to the documentation.
