Local checks

A local check can be written in any programming language supported by the target host. The script must be constructed so that each check produces a status line consisting of four parts. Here is an example:

The four parts are separated by blanks and have the following meanings:

There must always be exactly one space (ASCII 0x20) between the four parts of this output. Within the status details, any spaces can be used in any order.

If you are unsure about a possible output, you can simply test it by writing a small script with the echo command. Insert the output you want to test into the echo command. Our example uses double quotation marks on the outside, as variables inside (environment variables and those set in the script) are evaluated. As a result, you must enclose the quotation marks for the service name with \ so that these characters are not interpreted by the shell as the end and beginning of a string (and thus removed from the output):

For Windows hosts, such a script will look very similar to this:

Both scripts produce the same result in the output:

For Checkmk only this output is relevant, not how you created this output.

By the way — you can write any number of outputs in a script. Each output line will have its own service created in Checkmk. Therefore, no newline characters are allowed in the output — unless they are masked, for example for a multi-line output in Checkmk.

How it can be checked whether the local script will be correctly invoked by the agent can be seen in the Error analysis.

Once the script has been written it can be distributed to the appropriate hosts. The path used will depend on the operating system. A list of path names can be found in Files and directories below.

Don’t forget to make the script executable on Unix-like systems. The path shown in this example is for Linux (agent package with default settings):

If you use the Agent Bakery, the script can be distributed with a rule-based procedure. More on rule-creation can be found in the chapter Distribution via the Agent Bakery.

At every invocation of the Checkmk agent the local check contained in the script will also be executed and appended to the agent’s output. The service discovery also functions automatically like with other services:

Once the service has been added to the monitoring and the changes have been activated, the implementation of the self-created service with the aid of a local check will be complete. Should a problem arise during the service discovery, the Error analysis can be of help.

With a local script metrics can also be set. To enable them the value of the check must always return P. Then the state will be calculated by Checkmk.

The general syntax for this data is as follows:

where value is the current value, warn and crit set the (upper) thresholds, and min and max fix the range of values — for example like this:

The values are separated with a semicolon. All values except value are optional. If a value is not required, the field remains empty or is omitted at the end, as in the following for warn, crit and max:

Note: In the commercial editions the values for min and max can indeed be set — but only for compatibility reasons. Limiting the associated graph to a certain range of values has no effect in the commercial editions.

You should take special care when choosing the identifier of this metric - called metricname in the example here. We recommend prefixing the identifiers to prevent overlap with metrics already present in Checkmk.

So, for example, instead of simply calling a metric that represents the number of currently waiting requests in a queue you are monitoring, 'current', we recommend a clearer identifier with a prefix - such as: mycompany_current_requests.

If you were to choose an identifier here that already exists in Checkmk, the representation of your metrics in graphs would be overwritten with the definitions that already exist.

Of course, you can also reuse an existing metric from Checkmk intentionally. So, for a metric for an electrical current you could simply use the identifier current in your local check. In case of doubt, however, you have to look up the definition of this metric in ~/lib/python3/cmk/gui/plugins/metric by yourself.

You can also have several metrics output. These are then separated by the 'pipe' character |, for example like this:

Attention: On Windows hosts you have to prefix a caret (^) to the pipe in the script, so that it does not get interpreted.

A complete output with two metrics will look like this:

After you have also included the new service in the monitoring, you will see the text for the status detail in the Summary field in the service list. After clicking on the service, the page with the service details is displayed. The metrics are shown in the Details field and below this you will see the service graphs automatically generated by Checkmk:

In the previous chapters, you learned how to set threshold values for metrics and how to display them in the graphs. The next obvious step is to use these thresholds for a dynamic calculation of the service state. Checkmk provides exactly these options for extending a local check.

If you pass the letter P instead of a number in the first field of the output that determines the state, the service’s status will be calculated on the basis of the threshold as provided.

An output will then look like this:

… and the display in a service view like this:

The display differs in two points from the one that we saw earlier:

Some parameters have not only an upper threshold but also a lower threshold. An example is humidity. For such cases the local check has the option of passing two threshold values each for the states WARN and CRIT. They are separated by a colon and represent the lower and the upper threshold value respectively.

In the general syntax, it looks like this:

… and in the example like this:

… and in the display of a service view like this:

If you are only concerned with lower thresholds, leave out the upper threshold fields:

With this output, you specify that the service should become WARN if the value is less than 40 and CRIT if it is less than 30: thus, at the specified value of 37, the service will get the WARN state.

The option to spread an output over multiple lines is also available. Because Checkmk runs under Linux you can work with the Escape sequence '\n' in order to force a line-break. Even if due to the scripting language the backslash itself needs to be escaped, it will be correctly interpreted by Checkmk:

In the service’s details these additional lines will be visible under the Summary:

Under Linux or another Unix-like operating system, any plug-in can be executed asynchronously. For a local check, the necessary configuration is very similar to that of a plug-in. To do this, create a subdirectory called the number of seconds you want the output to be cached and put your script in that subdirectory.

In the following example, the local check will be executed only every 10 minutes (600 seconds):

The cached data is written to a cache directory.

For a service that provides cached data, the cache-specific information is added to the service view:

Under Windows, the configuration is also analogous to that of a agent plug-in. Instead of using a special subdirectory as with Linux & Co, the options are set in a configuration file:

As you can see above, under Windows you can configure the asynchronous execution (with async) and the time interval (with cache_age) separately.

Alternatively, on Windows you can also do the configuration in the Agent Bakery.

If you run into problems with a self-written script, you should check the following potential error sources:

If the script itself is correct, the agent can be run on the host. With Unix-like operating systems such as Linux, BSD, etc., the command below is available. With the -A option the number of additional lines to be displayed following a hit can be specified. This number can be customized to suit the number of expected output lines:

In the last line, you can recognize a cached service by the preceding cached information with the current Unix time and the execution interval in seconds.

Under Windows, you can achieve a very similar result with PowerShell and the Select-String 'cmdlet' as with the grep command under Linux. In the following command, the two digits behind the Context parameter determine how many lines are to be output before and after the hit:

As a last step the processing of the script output can also be tested on the Checkmk server with the cmk command — once for the service discovery:

… and also the processing of the service output with a similar command:

For both commands we have shortened the output by lines not relevant for this topic.

Alternatively, you can open the service list of the host in the monitoring, navigate to the service Check_MK and its column Icons. There you can choose the menu entry Download agent output to retrieve a text file containing the complete agent output.

If there are errors in a local check, Checkmk will identify them in the service output. This applies as well for erroneous metrics, for false or incomplete information in the script output, or an invalid status. These error messages should aid in quickly identifying errors in a script.

Cached data of individual sections, including the local section, is stored here and appended to the agent again with each execution, as long as the data is valid.