Retrieving status data via Livestatus

Access to the Livestatus is made over a Unix-Socket using the Livestatus Query Language (LQL). It’s syntax is based on HTTP.

Via the command line there are a number of ways of accessing the interface. One possibility is to use the printf and unixcat commands to send an instruction to the socket. The unixcat tool is already included in Checkmk for the instance user. Important: all inputs to the socket are case-sensitive so this must always be observed:

The interface expects all commands and headers in a separate row. You can mark such a line break with \n. As an alternative to the command above, you can also use the lq script command, which saves you a bit of work by auto-completing some fields when entering:

Or you can start the interactive entry stream and enter the command followed by the header. With a blank line you execute the command with its header, and with a further line the socket access is ended. Note that in the example, everything before the blank line belongs to the command, and everything between the first and second blank lines is the response:

The following examples are always executed with the lq command – in the direct form when the query is short, and as an entry stream for longer queries.

Since Checkmk is based very heavily on Python, scripts in this language are practical. The following script can be used as a basis for an access to the Livestatus socket:

Checkmk provides an API for the Python, Perl and C++ programming languages, which simplifies the access to Livestatus. An example code is available for each language which explains its use. The paths to these examples can be found in the chapter Files and directories.

In the examples we have seen so far, ALL information for ALL hosts has been queried. In practice however, one will probably only require specific columns. With the Columns header that has already been mentioned the output can be limited to this column. The individual column names will be separated by a simple blank character.

As can be seen, in a line the individual values are separated by a semicolon.

Important: If using these headers the header will be suppressed in the output. This can be re-inserted in the output with the ColumnHeaders header.

To limit the query to specific lines, the columns can be filtered for specified contents. If only services with a specific status are to be searched for, this can be achieved with a filter:

In the example all services with a CRIT status will be searched-for, and the host name, the service description and its status will be output. Such filters can of course be combined, and restricted to those services with a CRIT status, and which have not yet been acknowledged:

As can be seen, one can also filter by columns which are not listed in Columns.

You can use the header OrderBy to sort the output according to the contents of any column or even the values of any dictionary key. This allows you to create an alphabetical list of all hosts, for example:

Implicitly, the output is sorted in ascending order (asc) as expected. However, you can also specify the desired sorting explicitly and reverse it by specifying desc`:

With slightly longer queries and sorting by, for example, performance data, interesting ranking lists can be created. In the following example, the hosts are displayed in descending order of their uptime:

Some columns in a table return not just a single value, rather a whole list of them. So that such a list can be effectively searched, in these cases the operators have another function. A complete list of the operators can be found in Operators for lists. So for example, the operator >= has the function ‘contains’. With this you could, for example, search for specific contacts:

As can be seen in the above example, the contacts will be listed, separated by commas, in the contacts column. This allows them to be clearly distinguished as not being the start of another column. A special feature of the equality operator is that it checks whether a list is empty:

Several filters have earlier already been combined. It would seem to be intuitive that the data must pass through all filters in order to be shown. The filters will thus be linked by the logical operation and. To link particular filters with a logical or, at the end of the filter string code an or: followed by an integer. The counter specifies how many of the last lines may be combined with an or. In this way groups can be formed and combined as required. The following is a simple example. Here two filters are combined so that all services which have either the status WARN or UNKNOWN will be shown:

The result from a combination can also be negated, or groups can in turn be combined into other groups. In the example, all services are shown whose status is not OK, and whose description either begins with Filesystem, or who have a status other than UNKNOWN:

The output format can be specified in two ways. One method is to redefine the separators used in the standard output. The other method is to output conforming to Python or JSON formats.

There will be situations in which you have no interest in the status of a single service or group of services. Far more important is the number of services with a current WARN status, or the number of monitored data bases. Livestatus is able to generate and output statistics with Stats.

The Overview receives its data by retrieving statistics for hosts, services and events through Livestatus and displaying them in Checkmk’s interface. With direct access to Livestatus you can produce your own summary:

By the way, such statistics can be combined with all filters.

Statistics can also be combined with and/or. The headers are then called StatsAnd or StatsOr. Use StatsNegate if the output should be reversed. In the example the total number of hosts will be output (the initial Stats), and in addition the output will include the count of hosts marked as stale and which are also not listed in a Downtime (Stats 2 and 3 are linked with a logical 'AND'):

Do not be confused by the various options for combining the results from filters and statistics. While all hosts meeting the conditions will be output using the Filter header, with statistics the output will be the sum of how often the Stats filter applies.

It is also possible to perform calculations on values and, for example, output an average value or a maximum value. A complete list of all of the possible operators can be found here.

In the following example the output will list the average, minimum and maximum times a host’s check plug-ins require for calculating a status:

Calculations with metrics are handled in a somewhat special way. Here as well, all of the Stats-header functions are available for use. These are however applied individually to all of a service’s metrics. As an example, in the following example the metrics from a host group’s CPU-usage will be added together:

With the table log in Livestatus you have a direct access to the core’s monitoring history, so that using the LQL you can conveniently filter for particular events. The availability tables, for example, will be generated with the help of these tables. In order to enhance the overview and to restrict a query thematically, you have access to the following log classes:

Just by using these log classes you can already restrict which type of entry should be shown very well. The time range taken into account in the query will additionally be restricted. This is important since otherwise the instance’s complete history will be searched – which could logically apply a strong brake on the system due to the flood of information.

A further sensible restriction of the output are the (Columns) which are to be shown for an entry. In example below we will search for all notifications that have been logged in the last hour:

Important: Ensure that in the entry stream’s interactive mode none the of variables as used in the example can be used, and always restrict the queries to a time range.

It is possible to influence the rotation of the files, and their maximum sizes. You can additionally specify how many lines of a file should be read in before Checkmk interrupts. All of this can affect the performance of your queries, depending on the instance’s construction. The following three parameters are available which can be found and customized in Setup > General > Global Settings > Monitoring Core:

To access Livestatus via the network, you can connect the Unix socket of the live status process to a TCP port. This way you can execute scripts on remote machines and collect the data directly from where they should be processed.

When a site is turned off, access via TCP can be enabled with the omd command:

Once the site has been started, Livestatus via TCP is usually active on the default port 6557. For Checkmk servers with multiple sites using Livestatus via TCP, the next higher unused port is chosen.

All settings such as port and authorized IP addresses can be configured via omd config. Alternatively these settings can be made in the setup. In Checkmk the SSL encryption of Livestatus communication is enabled by default:

If access to Livestatus from outside your local network is required, access protection based on IP addresses alone may not be practical. The easiest way to gain authenticated access here is to use the Secure Shell.

With SSH, you have the ability to pass a command that will be executed on the remote server:

Alternatively, you can forward the Livestatus port to the host you are currently working on via an SSH tunnel:

If the connection has been established, in a second console session you can test whether access with openssl s_client is possible:

If this test is successful, any script you have written for direct Livestatus network access can be used on localhost.

Livestatus can not only be used for data queries, but also for issuing commands directly to the core (CMC or Nagios). A correct command always includes a time stamp – this can in fact be anything required. Because it will additionally be used in the Logs to track the time of the processing however, it is sensible to enter the time as precisely as possible. Commands with a missing time stamp will be discarded, without issuing an error message, and with only a simple entry in the cmc.log!

So that the time stamp can be as precise as possible, it is recommended to not set the command in the input stream, but rather to issue it directly. In such a situation there is also access to variables and the actual current time can be provided:

This format works with both the Nagios-Core in Checkmk Raw and with the CMC in the commercial editions. In the two cores the commands only partly-overlap however. A complete list of the commands for the Nagios-Core can be found directly on the Nagios website. The commands available for the CMC can be found in the Command reference.

In the list of the commands the syntax is in the following form:

As you have learned, Checkmk uses a much simpler format for issuing commands. To make the Nagios format compatible with Checkmk, you simply need the command, the time stamp, and where applicable, the variables: