Local checks

A local check can be written in any programming language supported by the target host. The script must be constructed so that each check produces a status line consisting of four parts. Here is an example:

The four parts are separated by blanks and have the following meanings:

There must always be a blank character between the individual parts of the output and the first text of the detailed status. Everything following will then count as status detail, which is why blank characters are allowed.

If there is uncertainty about a possible output, it can be simply tested by writing a small script with the echo command. Insert the output to be tested into the echo command. Make sure to mask the quotes for the service name with \ so that these characters are not interpreted by the echo command:

For Windows hosts, such a script will look very similar to this:

Both scripts lead to the same result in the output:

For Checkmk only this output is relevant, not how you created this output.

By the way — you can write any number of outputs in a script. Each output line will have its own service created in Checkmk. Therefore, no newline characters are allowed in the output — unless they are masked, for example for a multiline output in Checkmk.

How it can be checked whether the local script will be correctly invoked by the agent can be seen in the Error analysis.

Once the script has been written it can be distributed to the appropriate hosts. The path used will depend on the operating system. A list of path names can be found in Files and directories below.

Don’t forget to make the script executable on unix-type systems. The path shown in this example is for Linux:

If you use the Agent Bakery, the script can be distributed with a rules-based procedure. More on rule-creation can be found in the chapter Distribution via the Agent Bakery.

At every invocation of the Checkmk agent the local check contained in the script will also be executed and appended to the agent’s output. The Service discovery also functions automatically like with other services:

Once the service has been added to the monitoring and the changes have been activated, the implementation of the self-created service with the aid of a local check will be complete. Should a problem arise during the service discovery, the Error analysis can be of help.

With a local script metrics can also be set. To enable them the value of the check must always return P. Then the state will be calculated by Checkmk.

The general syntax for this data is as follows:

where value is the current value, warn and crit set the (upper) thresholds, and min and max fix the range of values — for example like this:

The values are separated with a semicolon. All values except value are optional. If a value is not required, the field remains empty or is omitted at the end, as in the following for warn, crit and max:

Note: In the Checkmk Enterprise Editions the values for min and max can indeed be set — but only for compatibility reasons. Limiting the associated graph to a certain range of values has no effect in the Enterprise Editions.

You should take special care when choosing the identifier of this metric - called metricname in the example here. We recommend prefixing the identifiers to prevent overlap with metrics already present in Checkmk.

So, for example, instead of simply calling a metric that represents the number of currently waiting requests in a queue you are monitoring, 'current', we recommend a clearer identifier with a prefix - such as: mycompany_current_requests.

If you were to choose an identifier here that already exists in Checkmk, the representation of your metrics in graphs would be overwritten with the definitions that already exist.

Of course, you can also reuse an existing metric from Checkmk intentionally. So, for a metric for an electrical current you could simply use the identifier current in your local check. In case of doubt, however, you have to look up the definition of this metric in ~/lib/python3/cmk/gui/plugins/metric by yourself.

You can also have several metrics output. These are then separated by the 'pipe' character |, for example like this:

Attention: On Windows hosts you have to prepend a caret (^) to the pipe in the script, so that it does not get interpreted.

A complete output with two metrics will look like this:

After you have also included the new service in the monitoring, you will see the text for the status detail in the Summary field in the service list. After clicking on the service, the page with the service details is displayed. The metrics are shown in the Details field and below this you will see the service graphs automatically generated by Checkmk:

In the previous chapters, you learned how to set threshold values for metrics and how to display them in the graphs. The next obvious step is to use these thresholds for a dynamic calculation of the service state. Checkmk provides exactly these options for extending a local check.

If you pass the letter P instead of a number in the first field of the output that determines the state, the service’s status will be calculated on the basis of the threshold as provided.

An output will then look like this:

… and the display in a service view like this:

The display differs in two points from the one that we saw earlier:

Some parameters have not only an upper threshold but also a lower threshold. An example is humidity. For such cases the local check has the option of passing two threshold values each for the states WARN and CRIT. They are separated by a colon and represent the lower and the upper threshold value respectively.

In the general syntax, it looks like this:

… and in the example like this:

… and in the display of a service view like this:

If you are only concerned with lower thresholds, leave out the upper threshold fields:

With this output, you specify that the service should become WARN if the value is less than 40 and CRIT if it is less than 30: thus, at the specified value of 37, the service will get the WARN state.

The option to spread an output over multiple lines is also available. Because Checkmk runs under Linux you can work with the Escape sequence '\n' in order to force a line-break. Even if due to the scripting language the backslash itself needs to be escaped, it will be correctly interpreted by Checkmk:

In the service’s details these additional lines will be visible under the Summary:

The output of local checks, like that of agent plug-ins, can be cached. This can be necessary if a script has a longer processing time. Such a script is then executed asynchronously and only in a defined time interval and the last output is cached. If the agent is queried again before the time expires, it uses this cache for the local check and returns it in the agent output.

Note: Caching is only available for AIX, FreeBSD, Linux, OpenWRT and Windows.

If you run into problems with a self-written script, you should check the following potential error sources:

If the script itself is correct, the agent can be run on the host. With unix-type operating systems such as Linux, BSD, etc., the command below is available. With the -A option the number of additional lines to be displayed following a hit can be specified. This number can be customized to suit the number of expected output lines:

In the last line, you can recognize a cached service by the preceding cache information with the current Unix time and the execution interval in seconds.

Under Windows, you can achieve a very similar result with PowerShell and the Select-String 'cmdlet' as with the grep command under Linux. In the following command, the two digits behind the Context parameter determine how many lines are to be output before and after the hit:

As a last step the processing of the script output can also be tested on the Checkmk server with the cmk command — once for the service discovery:

… and also the processing of the service output with a similar command:

For both commands we have shortened the output by lines not relevant for this topic.

If there are errors in a local check, Checkmk will identify them in the service output. This applies as well for erroneous metrics, for false or incomplete information in the script output, or an invalid status. These error messages should aid in quickly identifying errors in a script.

Cached data of individual sections, including the local section, is stored here and appended to the agent again with each execution, as long as the data is valid.