Checkmk on the command line

When administering Checkmk, with a few exceptions you need never work as the root-user. In this article we will generally assume that you are logged in as an instance user. That is done with, e.g.:

It is also possible to make a direct SSH-login to an instance without a detour via root. Since the instance user is a ‘completely normal’ Linux user, you must simply assign a password for this (which requires root-permissions, once only, for the configuration):

Afterwards an SSH-login directly from another computer should be possible (Windows-users preferably use PuTTY for this). From Linux this login is simply performed in the command line using the ssh command:

At the first login a ‘warning’ regarding an unknown host key will probably be received. When you are certain that in this brief moment no attacker has taken over your operating system’s IP-address, you can simply verify it with yes.

You can also work with the command line on the Checkmk-appliance. How that is done is explained in its own article.

So that as few problems as possible arise, particularly as a result of individual distributions or differing operating system configurations, the Checkmk-system ensures that the instance user – and likewise all of the monitoring’s processes – always have a clearly defined environment. Along with the home directory and the permissions, the environment variables (Environment) play an important role.

Among other things, when logging in as an instance user the following variables will be set or modified. These variables are available for use in all processes running within the instance. This also applies to scripts that are indirectly invoked by these processes (for example, a user’s own notification scripts).

With the env command you can output all of the environment variables – adding |sort to this command arranges the list a bit more clearly:

Under Linux the environment is an attribute of a process. Every process has its own variables, which it automatically passes on to subprocesses. These start initially with the same, inherited variables, but can also alter them.

With the env command you can always only view the the current shell’s environment. If you suspect there is an error in a particular process’s environment, with a small trick you can nonetheless output a listing of its environment. For this you only need the process-ID (PID). You can identify this with, e.g., ps ax, pstree -p or top. With this you can then access the process’s environ file directly via the /proc file system. Here as an example is a suitable command for the PID 13222:

If you require custom variables for your own scripts or other software to be run in the instance, store them in the etc/environment file which has been specially created for this purpose. All variables defined here will be available everywhere within the instance:

If you wish to customise your shell (Prompt or other things), you can perform this as usual in the .bashrc file. Environment variables nonetheless belong to etc/environment, so that they are certain to be available to all processes.

There is also nothing to prevent you having your own .vimrc file if you like working with VIM.

The following graphic shows the most important directories in a Checkmk-Installation, and as an example an instance named mysite which uses the Checkmk-Version 1.6.0p18:

The basis for this structure is provided by the /omd directory. Without exception, all of the files for Checkmk are found here. /omd is in fact a symbolic link to /opt/omd, while the actual physical data is located in /opt – but all data paths in Checkmk always use /omd.

Important is the separation of data (highlighted yellow) and software (blue). The instance’s data is found in /omd/sites, and the installed software in /omd/versions.

Like every Linux user, the instance user also has a home directory, which we refer to as the instance directory. If your instance is named mysite it will be found in /omd/sites/mysite. As usual in Linux the shell abreviates the its own home directory with a tilde (~) (or swung dash). Since immediately following a login you will actually be in this directory, the tilde appears automatically in the input prompt:

Subdirectories of the instance directory are shown relative to the tilde:

A number of subdirectories are located within the instance directory, these can be listed with ll:

As can be seen, the directories bin, include, lib, share and version are symbolic links. The rest are ‘normal’ directories. This mirrors the separation of software and and data as explained above. The software directory must be accessible as a subdirectory in the instance, but it is physically located in /omd/versions, and can also possibly be used by other instances.

The software directories, as usual under Linux, belong to root and thus may not be altered by an instance user. The following subdirectories are present – those in the example are physically located within the /omd/versions/1.6.0p18.cee, and they are accessible via symbolic links from the instance directory:

The version/ symbolic link is a ‘intermediate stop’ and serves as a relay point for the version used by the instance. During a software update this will be switched from the old to the new version. Nonetheless, please do not attempt to perform an update manually by altering the link, since an update requires a number of other further steps – which will fail.

The actual data for an instance is found in the remaining subdirectories in the instance directory. Without exception, these belong to the instance user. The instance directory itself is also included. Checkmk stores nothing apart from the directories listed there. You can create your own files and directories without problem here, in which tests, downloaded data, copies of log files, etc. can be kept as desired.

The following directories have been predefined:

As just shown in the above table, the local directory with its numerous subdirectories is intended for your own extensions. In a new instance, all of the directories in local/ are initially empty.

With the practical tree command you can quickly get an overview of the structure of local. The -L 3 option restricts the depth to 3:

All of the directories in the lowest level are actively integrated in the software. A file stored here will be treated in the same way as if it was in the directory with the same name within /omd/versions/…​ (or respectively, in the logical path from the instance under bin, lib or share).

Example: In the instance, executable programs will be searched for in bin and in local/bin.

Here it applies that in the case of identical names the file in local always has priority. This enables modification of the software without the need to change installation files in /omd/versions/. The precedure is simple:

Regarding point 3 above, if it is not known exactly which service to which the change applies, simply restart the complete instance with omd restart.

In Checkmk – as already-described – the log files are stored in the file directory var/. All components of the relevant log file can be found there:

Via the Global Settings on the web interface the comprehensiveness of the data to be recorded in the log files can be easily configured:

Alternatively it is of course possible to also customise the LogLevel on the global.mk file’s command line. This is in the directory for configuration files. Specify the entries if they are not already present:

For the cmc_log_levls the amount of logged data increases with the incrementation of the count. There are 8 levels ranging from 0 for Emergency to 7 which stands for Debug.

For notification_log you can choose between the values 10 for Full dump of all variables and command, 15 for Normal logging and 20 for minimal logging.

The alert_logging can be set to 10 for Full dump of all variables or 20 for Normal logging.

With the three values 'terse', 'full' and None for cmc_log_rrdcreation you can decide whether the creation of RRDs should be logged and how.

The level for the web interface log can be altered as required here:

The amount of logged data increases inversely as the count decreases. The lowest Loglevel is 50 (Critical) and while the most data will be logged at 10 which represents Debug.

The LogLevel for the Liveproxydaemon is set in the following file. The syntax is the same as with the web interface log:

Important: Log files can quickly become very large if a high level has been set. It is generally advisable to use such settings for a 'temporary' customisation, as an aid in problem identification for example.

The cmk command is actually an abreviation of check_mk, which was introduced to make typing the command easier. The command includes a built-in online help, that can as usual be called up with --help:

A number of options always work – regardless of the mode with which the command is executed:

The Checkmk Enterprise Editions utilise the Checkmk Micro Core (CMC) as its monitoring core, the Checkmk Raw Edition uses Nagios. An important task for the cmk is the generation of a configuration file that is readable for the core, and which contains all of the configured hosts, services, contacts, contact groups, time periods, etc. On the basis of this information the core knows which checks are to be executed and which objects it should provide using the GUI’s Livestatus.

For Nagios as well as for the CMC, it is fundamental that the number of hosts, services and other objects always remains static during the operation, and that this number can only be altered through the generation of a new configuration, followed by a reloading of the core. With Nagios a restart of the core is also needed. The CMC has a very efficient function for the reloading of its configuration during active processing.

The following table highlights important differences between the configurations of both cores:

Regenerating the configuration is always necessary if the contents of the configuration file in etc/check_mk/conf.d, or automatically-detected services in var/check_mk/autochecks have been modified. WATO keeps a record of such changes and highlights them in the GUI. Should you ‘bypass’ WATO by modifying the configuration manually or with a script, you will also need to attend to the activation manually. The following commands serve this function:

To summarise: If you want to customise a Checkmk-configuration and activate the changes, in Nagios you will subsequently require:

And with the CMC:

A second mode in Checkmk deals with the execution of a host’s Checkmk-based checks. With this you can allow all automatically detected, and also manually configured services, to be immediately checked, without needing to bother yourself with the monitoring core or the GUI. Simply enter the cmk command and the name of a host configured in the monitoring directly. Furthermore, you should always add both of the following options:

Further tips:

The following options influence the command:

An automatic service discovery can be started with cmk -I or cmk -II on the command line, and by specifying one or more hosts:

There are two modes for this:

All of the applicable details for this theme can be found in the relevant chapter in the article on the services.

The cmk command has a number of modes that are useful generally for diagnoses and troubleshooting. Here is an overview:

In the following section we will show how the commands can be used. The examples are mostly shown in an abreviated form.

WATO is a great web-based configuration tool. There are however many reasons to prefer a configuration with text data in the good, old Linux tradition. If you are of the same opinion there is some good news: Checkmk can be completely configured using text data. And since WATO does no more than process (this same) text data, this is not even an either/or situation.

If you are expecting a comprehensive compendium covering the exact structure of all of the configuration files used by Checkmk, we will unfortunately have to disappoint you here. The complexity and diversity contained in the configuration files is simply too much to describe completely in a handbook.

The following example shows an entire completed parameter set for the Check plug-in which monitors file systems in Checkmk. Because of the many parameters, the screenshot is divided into two parts, and set in lower-case characters:

The corresponding passage in the configuration file looks like this (somewhat more nicely formatted):

As can be seen, here there are no fewer than 14 different parameters, each with its own individual logic. Some are configured using floating-point numbers, (0.8), some with integers (24), some with keywords ('onlow'), some with boolean values (True), and others using tuples to code various combinations of these ((5.0, 10.0)).

This is just one example from over 1,000 plug-ins. And there are of course other configurations possible as check parameters: One only needs to think of time periods, event console rules, user profiles, and many more.

Of course that doesn’t mean you cannot use text data to generate a configuration! If you don’t yet know the exact syntax for your chosen configuration task, you only need the correct tool for it – and this tool we call WATO:

You thus only need to know in which file WATO writes.

There is a practical command for finding out which file WATO has just changed: find. By invoking ‘find’ with the following paramters you can find all files (-type f) under etc/ which have been altered within the last minute (-mmin -1):

The basis of a configuration is always the etc/check_mk directory. Below this is a subdivision into various domains, which generally apply to a specific service. At the same time each has a directory with the suffix .d, under which all files with the suffix .mk will be read automatically in alphabetic order. In some there will also be a main file which is read first of all. This is intended only for manual alteration, and is never modified by WATO.

The wato subdirectory is always found under the conf.d/-directory, e.g., etc/check_mk/conf.d/wato. WATO fundamentally only reads and writes here. The actual service reads the remaining files from conf.d if you have stored some manually-created files there. This means:

In purely formal terms, all of Checkmk’s configuration files are written in Python 2 syntax. There are two types of files:

The executable files can be recognised by their having variables which are substituted for assignments with values (=). The other files usually contain a Python-Dictionary which begin with an opening bracket '{'. Sometimes they are simple values.

If a non-ASCII character is required in a file (a German Umlaut (ä, ö, ü), for example), the following comment must be coded in the first or second line:

A syntax error will otherwise occur when reading the file. For further tips on Python syntax we recommend visiting a specialist site, for example: The Python Language Reference.