Integrating investigation-based alerts
Overview
In this section, learn about the notifications generated by investigation-based alerts to integrate them with other systems.
Investigation-based alerts return a set of objects matching the specified conditions either immediately or periodically, sending the result via email or, in the case of global alerts, optionally via the system log.
Create and configure an investigation-based alert using the Finder. The account used to create the alert has an influence on the mechanisms to notify it. If the account is properly configured with a valid email address, alerts associated with that account will send emails to the configured address. In addition to the configured email address, you may specify other recipients of the alert email in the dedicated space. If no email address has been configured for that particular account, at least one recipient has to be manually specified in the dedicated space.
Only those users with the appropriate profile setting (Allow system configuration) can create global alerts. Global alerts can be sent via email, as described above, and optionally via the system log.
Email integration of investigation-based alerts
Email is a proven, ubiquitous and mature technology, and thus a suitable means to integrate alert info into third-party software. Email is also easy to automate, since many programming languages have libraries available to send and receive email by means of standard email protocols such as SMTP, IMAP or POP.
Investigation-based alerts are sent via email in HTML form, using the UTF-8 charset and base64 transfer encoding. The subject of the message consists of the word Nexthink followed by a colon and then the name of the alert. The message content is composed of two HTML tables preceded by an embedded CSS snippet which defines the style of the two tables. The first HTML table displays some general information about the alert, whereas the second HTML table holds the result of the investigation associated to the alert, in the case of investigation-based alerts. If an investigation-based alert fails to execute, a message indicating the reason for the failure appears in the place of the results of the corresponding investigation.
In addition to the HTML table with the results of the investigation, the email of an investigation-based alert includes an attachment particularly well suited for integration. This is a compressed Comma Separated Values (CSV) file that holds the same results shown by the HTML, but in plain text. CSV files are understood by a great number of different tools and they are very easy to parse programmatically.
HTML info table
The HTML info table is composed of five fields which give general information about the context of the alert:
Source: name of the Engine that generated the alert.
User: name of the Finder account associated to the alert.
Name: the name of the alert itself.
Description: brief description of the alert, as displayed in the Finder.
Time or Period: For non-periodic (system or immediate) alerts, the time at which the alert was triggered is shown. In the case of periodic alerts, the period for which the alert was computed is displayed. In both cases, the time of the day or interval of time is expressed in the timezone of the associated user. The name of the timezone is displayed right after the corresponding time or period.
HTML results table
The results of an investigation-based alert are displayed in the form of a HTML table whose first row holds the names of the fields that were selected during the configuration of the alert. Up to a maximum of fifteen fields will be displayed in an email of an alert. If more than fifteen fields were selected when editing the investigation associated to the alert, only the first fifteen will appear in the email and the rest will be discarded. The CSS included in the mail makes the first row of the HTML results table to be highlighted, so the names of the selected fields appear as the headers of each column. Each subsequent row holds the values of the fields for every alerted object, that is, each row shows information about an object which met the conditions specified by the alert. The maximum number of alerted objects which can be displayed in the email of of an investigation-based alert is 250 objects. Therefore, a HTML results table may have a maximum of 251 rows, including the first row with the names of the fields. If more than 250 objects are alerted, a brief warning at the end of the email indicates that only partial results are shown.
Compressed CSV attachment
Although it is possible to parse the HTML results table for integrating its data into external software, the HTML tables of Nexthink alerts were mostly designed to be read by human beings. In addition to the HTML results table, however, the email sent by investigation-based alerts includes a compressed text attachment which is much more interesting for integration purposes. The attachment is a CSV file compressed with the well known Lempel-Ziv LZ77 algorithm whose name is always set to be "alert.zip". When uncompressed, the name of the file becomes "alert.csv". This attachment holds the same data as the HTML results table, with the advantage that its contents are easier to parse.
Once the attachment is uncompressed, the resulting CSV can be easily imported into third-party tools such as your favourite spreadsheet program.
Syslog integration
The system logging service, or syslog for short, is an alternative to email for integrating data coming from Nexthink alerts. Applications typically use the syslog to store messages that keep track of the activity of the application itself or that describe a situation that the application considers relevant. The syslog service is responsible for receiving these messages, assigning them a time-stamp and storing them in a timely manner.
In the Finder, you can select to send the results of a global investigation-based alert to the system log. Please note however that only those accounts with the right permissions are able to create global alerts.
Syslog configuration
The Nexthink appliance relies on the rsyslog package for writing to the system log. Many Linux distributions use rsyslog as their default service for system logging. If you are familiar with the configuration files of rsyslog, you may modify the format of alerts and of the Engine logs in general. The format of the configuration files of rsyslog is backwards compatible with the original syslog daemon. From this point on, we may refer to rsyslog as syslog when we talk about the service itself and not about a specific feature of rsyslog.
The configuration file for rsyslog is found in /etc/rsyslog.conf. For the sake of clarity, the specific modifications of Nexthink to the configuration of rsyslog are stored in a separate file, which is found in /etc/rsyslog.d/nx_rsyslog.conf. This file is applied to the main configuration file by means of an include directive in /etc/rsyslog.conf that reads all additional configuration files in the /etc/rsyslog.d folder
The part of the syslog configuration file /etc/rsyslog.d/nx_rsyslog.conf which is relevant for alerts is shown below:
The first line defines an output format for syslog messages by means of a template. The template is named RFC5424format because it follows the recommended format for syslog messages which is described in the most recent Internet standard about the syslog protocol: RFC 5424. The template defines the output to be composed of a priority number followed by the timestamp, the host name, the program name, the id of the process which issued the syslog message and the message itself. Once defined in this way, a template can be applied to one or several message filters.
For alerts, you can see that we declare two filters in the syslog configuration file, depending on the facility specified to log the alert. Both filters are instructed to write their output to the same file: /var/log/nexthink/alert.log. The minus sign before the file name is there to improve the performance of the syslog daemon. It indicates that syslog output to the file is buffered, so the syslog system will not directly write to the filesystem but to a buffer in memory and then really write to the disk once the buffer is full. The two filters however accept messages from different facilities. If the facility used is local5, rsyslog will use the default syslog output format. On the other hand, if the facility used is local6, rsyslog will use the output format defined by the template “RFC5424format” for every logged alert.
Alert format
We have seen that the format of an alert in the system log depends on the facility used to log the alert: local5 for default format and local6 for RFC 5424 format. The format of the message itself also depends on which facility is used by the Engine to log the alert. You can control the facility used to log alerts by means of a configuration parameter in the engine called legacy_alert_format in the syslog tag of the configuration file:
By default, the parameter is set to true in order to use the traditional alert format for syslog. Facility local5 is used in this default case. When local5 is used, the result of an alert is divided into two types of messages. The format of the first message is composed of the name of the alert and the number of rows that follow:
alert [n]
Then each row of the result is given in the following format:
alert | value1 | value2 | … |
where alert is again the name of the alert as saved with the Finder and valueN is the value that corresponds to the Nth field of the investigation associated to the alert. The messages are preceded by the timestamp and the default values set by syslog that depend on the default syslog configuration.
Example:
You may edit the file /var/nexthink/engine//etc/nxengine.xml manually to set the value of legacy alert format to false. If the value of this parameter is set to false, facility local6 is used for logging Engine messages. When local6 is used, the message generated for an alert combined with the template defined in the syslog configuration file has the following output format:
version timestamp hostname NX pid object [engine *(field="value")] alert [number/total]
where
pri: Priority of message. It is computed by first multiplying the number of the facility that sent the message by 8 and then adding the severity. The severity used by all log messages in the Engine is notice (5). Since the facility used is local6 (22) for non-legacy alerts, the priority is <181>.
version: Version of syslog protocol. We use version 1.
timestamp: High precision timestamp derived from RFC 3339.
hostname: Qualified name of the machine at the origin of the log message.
NX: This fixed value is the application name for the NEXThink Engine.
pid: Process ID of the Engine in the host machine.
object: Object category of the alarm investigation (e.g. source, user, destination, etc).
engine: Name given to the Engine in the server tag of the configuration file. Warning: this is not a valid SD-ID according to RFC 5424. We use it as a convention, but it may change in the future.
field: Name of the object parameter to display.
value: Value of the object parameter. The list of values is the actual result of the investigation.
alert: Name of the alert as saved with the Finder.
number/total: Number of the current row out of the total number of rows in the investigation result.
Example:
<181>1 2011-04-15T16:56:30.966693+02:00 Barahona NX 3286 source [DebugEngine name="QAXPRG" last_ip_address="192.168.0.44"] Last IP alert [1/1]
Known Limitations
In non-legacy alerts mode, the names of fields in the message of the logged alerts may not exactly match the names of the fields which where specified in the Finder when defining the alert. This is because the names used when generating the alert are the internal names of the fields as declared in the code of the Engine and not the names that you can see in the Finder. Usually, the two names are very similar if not equal, but do not rely on Finder names to parse alert results in the system log. The result of a periodic alert in the syslog does not specify the period for which the alert has been computed. Although the timestamps can give you a hint on this period, they do not provide a definitive answer.
The operations described in this article should only be performed by a Nexthink Engineer or a Nexthink Certified Partner.
If you need help or assistance, please contact your Nexthink Certified Partner.
RELATED TASKS
Last updated
