Configuring the system log

Overview

Syslog is a de facto standard solution for logging messages in UNIX-derived systems, such as the operating system of the Appliance. Programs use the syslog system call to send arbitrary messages to the syslog service. In addition to the message itself, two parameters are provided to the syslog system call: the facility and the level. The facility refers to the type of program that required the logging of a message. Facilities are named after typical UNIX services such as mail, ftp, or cron, subsystems such as the kernel, the printer, or the clock, and others are reserved for local use. The level indicates the importance or seriousness of the message. Possible values for the level are critical, warning, notice, etc.

The Nexthink components in the Appliance use the system log service to keep a record of significant occurrences, including:

• Audit trail events

• System alerts

• Investigation-based global alerts

• Internal state of the Engine

For writing to the system log, the Appliance relies on the rsyslog package, which has become the default logging service in many Linux distributions. Although it adds new advanced features, rsyslog still keeps backwards compatibility with the configuration files of the original syslog daemon. If you are familiar with the configuration of rsyslog, you may easily customize the output of the logs written by the Nexthink components and adapt them to your needs.

From this point on, we may refer to rsyslog as syslog when we talk about the logging service in general and not about specific features of rsyslog.

Default configuration and log files

The configuration file for rsyslog is found in /etc/rsyslog.conf. For the sake of clarity, the specific modifications of Nexthink to the configuration of rsyslog are stored in a separate file, which is found in /etc/rsyslog.d/nx_rsyslog.conf. This file is applied to the main configuration file by means of an include directive in /etc/rsyslog.conf that reads all additional configuration files in the /etc/rsyslog.d folder.

The default configuration of Nexthink dispatches log messages to different files depending on their content. Find these log files under /var/log/nexthink:

File	Purpose
alert.log	System and investigation-based global alerts Debug info from logger (rsyslog)
audit.log	Audit trail events
engine.log	Internal state of the Engine

By default, the Portal, the Engine, and the Web Console write their audit events to the audit.log file of the Appliance that hosts each one of them. Only the Engine writes to the alert.log and the engine.log files. In turn, the Portal does not use the syslog service to write information about its internal state, but its own logging tools. The internal logs of the Portal are found under /var/nexthink/portal/log.

The audit and alert logs are suitable for automatic processing, since their format is well-defined and stable. However, the format of the internal logs of the Engine is not guaranteed and may be subject to change. Therefore, do not rely on the contents of the Engine log for automating your processes.

Nexthink uses UTF-8 encoding for its log messages. Rsyslog preserves the encoding.

Configuration of alerts

In addition to email, you can use the system log as a notification mechanism for both system and global alerts. For global alerts, you need to enable syslog notification when creating the alert.

The part of the syslog configuration file /etc/rsyslog.d/nx_rsyslog.conf which is relevant for alerts is shown below:

Code

 $template RFC5424format,"<%pri%>1 %timestamp:::date-rfc3339% %hostname%
 %programname% %procid%%msg%\n"
 ...
 # alerts
 local5.=notice -/var/log/nexthink/alert.log;
 ...
 # alerts
 local6.=notice -/var/log/nexthink/alert.log; RFC5424format

The first line defines an output format for syslog messages by means of a template. The template is named RFC5424format because it follows the recommended format for syslog messages which is described in the most recent Internet standard about the syslog protocol: RFC 5424. The template defines the output to be composed of a priority number followed by the timestamp, the host name, the program name, the id of the process which issued the syslog message and the message itself. Once defined in this way, a template can be applied to one or several message filters.

For alerts, you can see that we declare two filters in the syslog configuration file, depending on the facility specified to log the alert. Both filters are instructed to write their output to the same file: /var/log/nexthink/alert.log. The minus sign before the file name is there to improve the performance of the syslog daemon. It indicates that syslog output to the file is buffered, so the syslog system will not directly write to the filesystem but to a buffer in memory and then really write to the disk once the buffer is full. The two filters however accept messages from different facilities. If the facility used is local5, rsyslog will use the default syslog output format. On the other hand, if the facility used is local6, rsyslog will use the output format defined by the template RFC5424format for every logged alert.

To choose between legacy (local5) or modern (local6) format for the log messages of global alerts, set the following parameter in the main section of the configuration file of the Engine (/var/nexthink/engine/01/etc/nxengine.xml):

Code

  <syslog>
   <legacy_alert_format>true</legacy_alert_format> 
 </syslog>

For details on the formatting of alerts, see the article on integrating alerts.

Customizing the syslog configuration

To preserve the standard log files and formats used by Nexthink, avoid directly editing the Nexthink configuration file for rsyslog /etc/rsyslog.d/nx_rsyslog.conf. Instead, to customize the destination and format of the logs, write your own formatting templates and rules (aka selectors) in a custom configuration file located here: /etc/nexthink/nx_rsyslog_custom.conf. The Nexthink configuration file for rsyslog includes the custom configuration file before declaring its own rules:

$IncludeConfig /etc/nexthink/nx_rsyslog_custom.conf

Custom rules are thus processed before standard Nexthink rules. Again, to preserve the default configuration and standard log files of Nexthink, do not write any rules in the custom configuration files that stop the processing of messages by the standard Nexthink rules.

Logging to a remote server

The syslog protocol lets you send log messages through the network to be consumed by syslog servers other than the local Appliance.

To send log messages to a remote syslog server, copy each line in the syslog configuration file of Nexthink that you want to modify to your custom configuration file and substitute the name of the log file for the name or IP address of the receiving server. The name of the server must be preceded by a single or a double at-sign (@ or @@), depending on whether you want to send the log messages via UDP or TCP, respectively. Follow the name or IP address of the remote server by a colon (:) and the port number where the server is listening for syslog messages. For example, to send different types of log messages to remote servers.

Remember to use either local5 or local6 entries in secondary Appliances, depending on the setting for the Engine legacy_alert_format to be true or false, respectively. For primary Appliances, recall that the Portal always uses syslog local5 facility and exclusively for audit events:

Code

# Send general log to a server listening to UDP port 514
local5.=debug;local5.=info;local5.=error @udp-server.example.com:514; nxFormat

# Send audit logs to a server listening to UDP port 514
local5.=warning @udp-server.example.com:514; nxAuditFormat

# Send alert logs to a server listening to TCP port 10514
local5.=notice @@tcp-server.example.com:10514;

Note that you do not have to choose between saving the logs in a file and send them to a remote server. It is possible to do both by repeating the same line in the syslog configuration changing the destination of the logs. Check the rsyslog documentation for options when sending log messages through the network, specially when using TCP.

Logging accesses to the CLI

Besides user access to the Nexthink components such as the Finder and the Portal, the access to the command line interface of the Appliance is an event of interest in the audit trail.

To filter the syslog messages related to accesses to the CLI of the Appliance and send them to a destination of your choice, specify the programs that control the command line inside conditional statements in your custom configuration file for syslog:

Code

# Log access to the CLI
if $programname == 'sshd' then -/var/log/nexthink/audit.log
if $programname == 'sudo' then @udp-server.example.com:514
if $programname == 'login' then @@tcp-server.example.com:10514

These programs control remote access (sshd) to the Appliance, logging in (login) to the Appliance, and execute as the superuser (sudo) in the Appliance. In the example above, each program is sending its output to a different destination, but you can send the output of all programs to the same destination.

Restarting the Engine and the syslog service

Restart the Engine if its configuration file required any change:

sudo systemctl restart nxengine@1

After any modification to the configuration files of syslog, restart the service for the changes to be effective:

sudo systemctl restart rsyslog

---

RELATED TASKS

• Creating an investigation-based alert

• Integrating alerts

• Examining the logs in the Portal

RELATED REFERENCES

• System alerts

• Audit trail

• Rsyslog (external link)