1. Introduction


Linux systems can be especially well monitored by Checkmk. This has less to do with the Checkmk development team ‘feeling at home’ with Linux — rather it has more to do with Linux being a very open system, and it providing numerous very well documented and easily queried interfaces which enable a comprehensive monitoring operation.

The installation of a monitoring agent is essential, since most of the interfaces are not in themselves accessible over a network. For this reason Checkmk has its own agent for the monitoring of Linux. This is easy to install and is economical in its use of system resources.

The Checkmk agent for Linux is…​

  • …​ minimalistic, since it requires little CPU, RAM and hard drive space.

  • …​ transparent, because it is a shell script, in which you can see which command it is executing.

  • …​ secure, since it permits no access from the network whatsoever.

The agent consists of a simple shell script that is installed in /usr/bin/check_mk_agent, and which invokes existing system commands sequentially in order to seek data for the monitoring. It routes its output using either xinetd or systemd to TCP-Port 6556, or it is alternatively called per SSH. Should you like neither of these procedures you can also implement your own methods for Checkmk to receive data from agents.

Monitoring of information that the standard agent doesn’t provide can be added in the form of agent plug-ins. In the Enterprise Editions you can bundle all settings and plug-ins together with the agent into an RPM or DEB packet using the Agent Bakery, this packet can then be installed with a single command and/or even be updated fully automatically.

2. Installation

2.1. Various options

Checkmk provides a number of ways to install a Linux agent — from the manual installation of its separate components, to a fully automated deployment. Some of these methods are only available in the Enterprise Editions:


Included as standard RPM/DEB-Packet

Simple installation of a standard agent with a manual configuration over configuration files



RPM/DEB-Packet from the Agent Bakery

Configuration via the GUI — individual configuration of each host is possible


automated updates

The packet from the Agent Bakery is initially installed manually or by using a script, and subsequently from there updated automatically.


Manual installation

The individual files, without a packet, are copied to the target system and then xinetd, systemd, SSH or a user’s own access method are manually installed.



2.2. Installation using a RPM/DEB-Package

When you are using the CRE Checkmk Raw Edition you will find the Linux packages via Setup > Agents > Linux. In the Enterprise Editions clicking Setup > Agents > Windows, Linux, Solaris, AIX will lead to the Agent Bakery. From here, you find the list of agent files by clicking Related > Linux, Solaris, AIX files.

agent linux agent files

Everything required is in the first section called Packaged Agents. Finished RPM and DEB packages for the installation of a Linux agent with standard settings can be found here.

Whether you require RPM or DEB depends on the Linux distribution on which the package is to be installed:

PackageFile extensionFor installation on



Red Hat Enterprise Linux, Fedora, CentOS, openSUSE, SLES, and derivatives of these



Debian, Ubuntu, and all other DEB-based distributions

Attention: in the Agents section you find the straight agent scripts for the manual installation to /usr/bin/. Only use these if the packaged agents do not function on your system, or if you wish to monitor Unix derivatives other than Linux.

Exotic Linux distributions

Additionally, if the Checkmk server only supports Enterprise-Linux distributions that are still supported by their producer, the Checkmk agent is in such a case much less demanding. It supports every Linux distribution – even ancient ‘dinosaurs’ on which a Version 2.4 core is running! It is however possible that not all plug-ins will run correctly in the agent, but the basic monitoring itself will function.

Installation with RPM

The installation of an agent with the RPM package is very simple:

  1. Download the agent packet with the *.rpm file extension.

  2. Copy it to the target system to be monitored with scp, WinSCP or similar.

  3. Install it on that system as root using the rpm -U command.

  4. Next install the xinetd packet from your distribution, if it is not already installed.

Here is an example of a package installation:

root@linux# rpm -U check-mk-agent-2.0.0p42-1.noarch.rpm

By the way, the -U stands for ‘Update’, but it can also perform an initial installation correctly. This means that the same command can also be used later for an update to a newer version.

Following the installation an agent is immediately active and can be queried through TCP Port 6556.

Installation with DEB

This procedure is the same as for RPM, only the commands are different. An installation and a later update of the agent are both performed using:

root@linux# dpkg -i check-mk-agent_2.0.0p42-1_all.deb

Get package via HTTP

Sometimes scp or WinSCP can be very laborious. The packet can also be loaded directly from the Checkmk server to the target system over HTTP. For this purpose the agent file downloads have deliberately been made possible without needing a login. The files after all contain no secrets. Anyone can download and install Checkmk themselves and then access the files.

The easiest method is to use wget. The URLs can be identified via the browser. If you know the package’s name you can simply assemble the URL name yourself. Then simply set /mysite/check_mk/agents/ before the file name:

root@linux# wget http://mycmkserver/mysite/check_mk/agents/check-mk-agent-2.0.0p42-1.noarch.rpm

RPM actually has an inbuilt wget — with this you can perform the download and the installation in a single action:

root@linux# rpm -U http://mycmkserver/mysite/check_mk/agents/check-mk-agent-2.0.0p42-1.noarch.rpm

Get package via REST API

The REST API in Checkmk provides the following ways to download agent packages as of release 2.0.0p18:

  • Download a baked agent by host name and operating system

  • Download a baked agent by agent hash and operating system

  • Download agents shipped with Checkmk

For example, the DEB package of the Linux agent included in Checkmk can be fetched as follows:

curl -OJG 'mycmkserver/mysite/check_mk/api/1.0/domain-types/agent/actions/download/invoke'; \
--data-urlencode 'os_type=linux_deb' \
-H 'accept: application/octet-stream' \
-H 'Authorization: Bearer automation myautomationsecret'

This is just a very simple example to demonstrate how this one specific endpoint works. As always, for details on how the agent download endpoints work, please refer to the API documentation available in Checkmk, which can be found via Help > APIs > REST API documentation.

2.3. Installation with the Agent Bakery

CEE The CEE Checkmk Enterprise Editions utilise the Agent Bakery via a Setup module to package individually-customized agents. This process will be described in the general chapter covering the agents.

Installation of the baked packets is performed exactly as described above.

2.4. Automatic updates


If you use the Agent Bakery, the agent’s automatic updates can set this up. This process is described in its own article.

2.5. Manual installation

The manual installation of an agent is rarely necessary, but is not very difficult. As well as the agent files from the site, you also require the Agents selection menu — here you will find the Checkmk Agent for Linux file:

agent linux agents manual

Load this file to the target system, and copy it to a directory that is executable for root. /usr/local/bin/ is very well suited for this, since it is found in the search path and has been conceived for the user’s own extensions. Here you can also work directly with wget:

root@linux# cd /usr/local/bin
root@linux# wget http://mycmkserver/mysite/check_mk/agents/check_mk_agent.linux
root@linux# mv check_mk_agent.linux check_mk_agent
root@linux# chmod 755 check_mk_agent

Please don’t forget the last two commands — these remove the .linux file extension and make the file executable. If everything has been done correctly the agent should now simply be an executable command which will generate its usual output. This also works if you are not in /usr/local/bin. The | head truncates everything after the 11th line:

root@linux# check_mk_agent | head
Version: 2.0.0p42
AgentOS: linux
Hostname: mycmkserver
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local

In the case of a very old distribution (published prior to 2010) which does not recognise the timeout command, load the small program waitmax from the agent page (in the Agents box) and install it likewise in /usr/local/bin. Both timeout and waitmax perform the same function — they force a timeout when executing a program:

root@linux# timeout --help
  or:  timeout [OPTION]
  Start COMMAND, and kill it if still running after DURATION.

Waitmax was developed as a Checkmk component at a time when timeout was not widely used. It has almost the same call syntax:

root@linux# waitmax --help
age: waitmax [-s SIGNUM] MAXTIME PROGRAM [ARGS...]

Execute PROGRAM as a subprocess. If PROGRAM does not exit before MAXTIME
seconds, it will be killed with SIGTERM or an alternative signal.

   -s, --signal SIGNUM   kill with SIGNUM on timeout
   -h, --help            this help
   -V, --version         show version an exit

Should you wish to configure or extend the agent, you will need to create the required directories yourself. The location for the three required directories is hard coded in variables that begin with MK_, and which will also be provided to the plug-ins over the environment:

root@linux# grep 'export MK_' check_mk_agent
export MK_LIBDIR="/usr/lib/check_mk_agent"
export MK_CONFDIR="/etc/check_mk"
export MK_VARDIR="/var/lib/check_mk_agent"

These three directories should be created (with the standard permissions 755):

root@linux# mkdir /usr/lib/check_mk_agent /etc/check_mk /var/lib/check_mk_agent

Should you want to change the file path, simply edit it in /usr/local/bin/check_mk_agent.

If you fundamentally want to call up the agents over SSH, a configuration for the xinetd is required, you just additionally need the SSH-configuration. How that is achieved will be described below.

The configuration per xinetd enables an access of the agent data via TCP Port 6556, and is the standard method in local networks. Install the xinetd package for this, and create the following file:

service check_mk
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/local/bin/check_mk_agent
        only_from      =
        disable        = no

Here enter the IP address for your Checkmk server that will be permitted to access the agent under only_from. Then only an activation is required and the agent will be ready:

root@linux# /etc/init.d/xinetd reload

If you’re using systemd, you should execute the following command instead:

root@linux# systemctl reload xinetd.service

3. Testing, and error diagnosis

3.1. Overview

Once the agent has been installed, the obvious question will surely be how to test whether everything has been done correctly. All of the possibilities that are available from the Checkmk server are described in the general chapter on the agents. There are of course further diagnostic possibilities when one is directly logged into the target system itself.

Since the ‘agent’ is basically nothing more than a simple program that obtains data from your system and outputs it as loosely-formatted text, you can also invoke it as a program, and in fact do so quite easily:

root@linux# check_mk_agent
Version: 2.0.0p42
AgentOS: linux
Hostname: myhost123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
udev              devtmpfs     8155492         4   8155488       1% /dev
tmpfs             tmpfs        1634036      1204   1632832       1% /run
/dev/sda5         ext4       226298268 176973752  37806104      83% /
none              tmpfs              4         0         4       0% /sys/fs/cgroup

Because the output can be rather long, less can also be very practical here (you can quit it with the 'Q' key):

root@linux# check_mk_agent | less

This output does not of course prove whether the agent is also accessible over the network. But in this way it can be tested whether all of the desired data is present in the output.

Incidentally — it is not essential to be root to be able to invoke the agent, however it is possible that the output could be missing some information which requires root-permissions to obtain (e.g., multipath information and the output from ethtool).

3.2. Debug mode

So that possible erroneus outputs from defective plug-ins or commands do not ‘corrupt’ the real production data, as a matter of principle the agent suppresses the standard error channel. If one is looking for a specific problem this can be reactivated by invoking the agent with a special debug mode. This is done with the -d option. In so doing all of the shell commands which the agent executes will be output.

So that you can work with less here, the standard output and error channels must be combined with 2>&1:

root@linux# check_mk_agent -d 2>&1 | less

4. Disable sections

The output of the Checkmk agent is divided into sections. Each of these sections contains related information and is usually simply the output of a diagnostic command. Sections always start with a section header. This is a line enclosed in <<< and >>>.

Except for Checkmk’s own sections, you can individually disable any of the 40+ sections that the agent generates by default. Specifically, this means that the corresponding commands will not be executed by the agent at all, possibly saving computation time. Other reasons for deactivation could be that you are simply not interested in certain information of a certain group or type of hosts, or that a certain host delivers incorrect values and you want to temporarily suspend the retrieval of this data.

As a user of one of the CEE Checkmk Enterprise Editions you can simply create a rule via Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Disabled sections (Linux agent), which will then be taken into account by the Agent Bakery.

agent linux disabled sections

In the rule you will then find a separate checkbox for each section that can be disabled. For each selected checkbox you will find — after the newly baked agent has been installed on the selected hosts — a separate entry in the file /etc/check_mk/exclude_sections.cfg. For example, if you were to select Running processes and Systemd services, the appropriate configuration file would look like the following:


Users of the CRE Checkmk Raw Edition can manually create the above /etc/check_mk/exclude_sections.cfg file and enter there the sections that should be disabled. All sections that can be disabled are listed in the file ~/share/check_mk/agents/cfg_examples/exclude_sections.cfg.

5. Integration of classic check plug-ins

5.1. Executing plug-ins over MRPE

If you have migrated your monitoring from a Nagios-based solution to Checkmk, it cannot be ruled out that you have existing check plug-ins of a standard form for which no counterpart in Checkmk (yet) exists. In most cases these will be self-written plug-ins in Perl or Shell.

The Checkmk agent offers a simple mechanism that still enables such plug-ins to be used: MK’s Remote Plugin Executor or MRPE for short. The name is intentionally an analogy to NRPE in Nagios, for which it performs the same function.

The MRPE is integral in agents and is configured with a simple text file which you yourself can save in /etc/check_mk/mrpe.cfg. In this file enter one plug-in call per line — together with the name that Checkmk should use for the service that is to be automatically created. Here is an example:

Foo_Application /usr/local/bin/check_foo -w 60 -c 80
Bar_Extender /usr/local/bin/check_bar -s -X -w 4:5

Note: The Nagios plugins may not be placed in the directory /usr/lib/check_mk_agent/plugins. This directory is reserved for the agent plug-ins. Apart from this directory, you are free to choose as long as the agent can find and run the plugins there.

If the agent is allowed to run locally, for each plug-in a new section will be found with the title <<<mrpe>>>, containing the name, exit code and output from the plug-in. This can be verified with the following practical grep-command:

root@linux# check_mk_agent | grep -A1 '^...mrpe'
(check_foo) Foo_Application 0 OK - Foo server up and running
(check_bar) Bar_Extender 1 WARN - Bar extender overload 6.012|bar_load=6.012

The 0 or 1 in the output stand for the plug-in’s exit codes and conform to the standard pattern: 0 = OK, 1 = WARN, 2 = CRIT and 3 = UNKNOWN.

The rest will be taken care of by Checkmk automatically. Once a service discovery has been performed for the host, the two new services will be shown as available. It will look like this:

agent linux mrpe checks

Incidentally: due to the file’s syntax the name is not permitted to contain blank characters. With the help of the same syntax as used in URLs, a space can be replaced by %20 (the ASCII-Code 32 for ‘space’ is Hexadecimal 20):

Foo%20Application /usr/local/bin/check_foo -w 60 -c 80
Bar%20Extender /usr/local/bin/check_bar -s -X -w 4:5

5.2. Asynchronous execution

Please note that all plug-ins running in mrpe.cfg will be executed synchronously and sequentially. The plug-ins should thus not have overly long runtimes. If a plug-in hangs, all following plug-ins will be delayed. This could lead to all of the agent’s retrievals under Checkmk entering a timeout, meaning that the host can no longer be reliably monitored.

If you really require longer running plug-ins, these should be converted to asynchronous processing, thus avoiding the problem described above. For this, define a time frame in seconds for which a calculated result will be valid — for example, 300 for five minutes. To achieve this, set the expression (interval=300) following the service name in mrpe.cfg:

Foo_Application (interval=300) /usr/local/bin/check_foo -w 60 -c 80
Bar_Extender /usr/local/bin/check_bar -s -X -w 4:5

This will have a number of effects:

  • The plug-in will be executed in a background process and will no longer hinder the agent’s processing.

  • Because the agent no longer waits for the execution, the result will be delivered at the next call of the agent.

  • The plug-in will not be run before the defined 300 seconds have elapsed. Until that time the old result will be recycled.

With this method tests that require a little more processing times, as well as longer intervals, can also be run without needing to make changes to the configuration on the Checkmk server.

5.3. MRPE with the Agent Bakery

CEE Users of the Enterprise Editions can also configure MRPE with the Agent Bakery. The rule set Setup > Agents > Windows, Linux Solaris, AIX > Agent Rules > Generic Options > Execute MRPE checks is responsible for this. There you can configure the same things as described above. The appropriate mrpe.cfg file will then be automatically generated by the Agent Bakery.

agent linux mrpe rule

Baking the plug-ins

Likewise the Check plug-ins themselves can be delivered with the package. With these the agent will be complete and will require no manual installation of additional files. This is all performed as follows:

  1. Create the directory local/share/check_mk/agents/custom on the Checkmk server

  2. Create a subdirectory, e.g. my_mrpe_plugins in the above directory

  3. Then create the subdirectory bin in this

  4. Copy your plug-ins to this directory

  5. Create a rule in Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Generic Options > Deploy custom files with agent.

  6. Select my_mrpe_plugins, save and bake it!

The Check plug-ins will be installed in your agent’s standard-bin-directory. By default this is /usr/bin. Then when configuring the MRPE-checks, use /usr/bin/check_foo instead of /usr/local/bin/check_foo.

6. Extending the agent with plug-ins

6.1. What are plug-ins?

The standard agent /usr/bin/check_mk_agent contains a whole series of sections which provide monitoring data for various checks which will then be found automatically by a service discovery. These include all of an operating system’s important monitorings.

Additionally, there is also the possibility of augmenting an agent with plug-ins. These are small scripts or programs which are invoked by an agent to include further sections with additional monitoring data. The Checkmk project delivers a whole series of such plug-ins, which — when correctly installed and configured — via a service discovery can provide new checks automatically.

Why are these plug-ins not simply built into the standard agent? For each plug-in one of the following reasons prevents such an integration:

  • The plug-in is coded in a programming language other than Shell and thus cannot be implemented inline (example: mk_logwatch).

  • The plug-in in any case requires a configuration without which it will not function (example: mk_oracle).

  • The plug-in is so specialised that most users will not require it (example: plesk_domains).

6.2. Manual installation of plug-ins

The plug-ins for Linux and UNIX included with the project are all located on the Checkmk server in share/check_mk/agents/plugins. These are additionally available via the download page for the agents in the Setup menu (as described at the beginning of this article) in the Plugins box:

agent linux files agent plugins

For all of our standard agent plug-ins there are also the matching check plug-ins which can evaluate the data and create services from these. They are already installed so that new services can be immediately discovered and configured.

Before installing a plug-in in an agent, please have a look at its associated file. Important tips for the correct use of the plug-in can often be found there.

The actual installation is then simple: simply save the file to /usr/lib/check_mk_agent/plugins. When doing this ensure that the file is executable. If not, use a chmod 755, otherwise the agent will not run the plug-in. Especially if the files are not transferred over scp, rather via HTTP from the download page, the execution permissions will be lost!

Once a plug-in is executable and in the correct directory, it will be invoked by the agent and a new section will be generated in the agent’s output. This section usually has the same name as the plug-in. Complex plug-ins (e.g., mk_oracle) in fact create a whole series of sections.

In older versions of the Checkmk agent the plug-in directory can be in another location. If case of uncertainty, or you don’t know if this applies in your setup, the directory can be identified with the following:

root@linux# grep MK_LIBDIR= /usr/bin/check_mk_agent
export MK_LIBDIR="/usr/lib/check_mk_agent"

6.3. Configuration of plug-ins

Some plug-ins require a configuration file in /etc/check_mk/ to be able to function. With others a configuration is optional and allows special features or customization. Others simply work as is. There are various sources of relevant information:

  • The documentation for the matching check plug-ins within your Checkmk site, which you find via Setup > Services > Catalog of check plugins

  • Comments in the plug-in itself (often very helpful!)

  • A relevant article in this User guide (e.g., on the monitoring of Oracle)

6.4. Asynchronous execution

Plug-ins can process asynchronously in the same way as with MRPE. This is very useful if the plug-ins have a very long runtime, and the acquired status data in any case does not need to be refreshed every minute.

An asynchronous execution is not configured with a file. Instead, create a subdirectory in plugins whose name is a numeric, representing a count in seconds. Plug-ins in this directory will not only be executed asynchronously, at the same time by specifying the second count a minimum waiting time is also specified before the plug-in can be run again. If the agent is again called before the specified time has elapsed, it will use cached data from the last run of the plug-in. In this way an interval longer than the typical one minute can in effect be configured for the plug-in.

The following example shows how by specifying a 5 minute interval the my_foo_plugin plug-in can be changed from synchronous to asynchronous execution:

root@linux# cd /usr/lib/check_mk_agent/plugins
root@linux# mkdir 300
root@linux# mv my_foo_plugin 300

Please note that a few plug-ins are set up to execute asynchronously by default. Among these is mk_oracle. Always install such plug-ins directly in /usr/lib/check_mk_agent/plugins!

6.5. Installing plug-ins using the Agent Bakery

CEE With the CEE Checkmk Enterprise Editions included plug-ins can be configured using the Agent Bakery. This not only takes care of the installation of the plug-ins themselves, but also for the correct generation of the configuration file should one be required.

Each plug-in is configured with an agent rule. The appropriate rule set can be found in Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Agent plugins:

agent linux rules agent plugins

6.6. Manual execution of plug-ins

Since agent plug-ins are executable programs, for testing and diagnosis they can be manually started. There are however plug-ins which require specific environment variables to be set by agents — so that they can find their configuration file, for example. Set these variables manually before execution:

root@linux# export MK_LIBDIR=/usr/lib/check_mk_agent
root@linux# export MK_CONFDIR=/etc/check_mk
root@linux# export MK_VARDIR=/var/lib/check_mk_agent
root@linux# /usr/lib/check_mk_agent/plugins/mk_foobar

Some plug-ins have special invocation options for debugging. Just have a look in the plug-in!

7. Security

7.1. Considerations

Security is an important criterion for any software — and monitoring may be no exception. Because the monitoring agent is installed on every server being monitored, a security problem here can have serious consequences.

For this reason emphasis is placed on security in the design of Checkmk, and from the earliest days of Checkmk security has been an unshakeable principle: An agent reads no data from the network — full stop. Consequently it is quite impossible that an attacker could sneak any type of command or script element over the monitoring port 6556.

This alone provides such a high level of security that most users do without additional measures in the LAN. If the system being monitored is only accessible over an insecure internet connection, then of course quite different precedures are necessary, and an encryption with SSH would certainly be the ideal first choice.

The Checkmk agent additionally includes an inbuilt encryption, which represents a good compromise between security and complexity. In the following section we will show all of the options for protection in detail.

7.2. Restriction of access over IP addresses

Even if an attacker is unable to execute commands, the monitoring data could be useful to him as among other info, the data includes a list of all processes running on the system. Is is therefore best that the data is accessible to nobody.


If the Checkmk agents are as usual enabled over the xinetd, it is very simple and effective to restrict the access to specific IP addresses — to those of the monitoring server of course. This is easily done, and was in fact to be seen in the example above:

service check_mk
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent
        only_from      =
        disable        = no

CEE In the Enterprise Editions Agent Bakery users can configure the permitted IP addresses via the Allowed agent access via IP address rule set. This rule set can be found via Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Generic Options.

An attacker can of course easily falsify their IP address and thus make a connection to an agent — but it is very unlikely that it will receive an answer since this goes to the genuine monitoring server. Or it really does receive an answer, but the Checkmk server looks down the pipe and will very quickly record an error.


Since most distributions have switched to systemd, which not only provides the tasks of the init system, but also provides network functionality, Linux distributors are working to get rid of the good old xinetd. The prepackaged Linux agent (not one from the Agent Bakery!) already installs itself with systemd instruments, if the target system is based on systemd and no xinetd is available.

When using systemd, it is of course also possible to restrict which machine or which IP may query the Checkmk agent. The systemd-counterpart to only_from known from xinetd is called IPAddressAllow. During its installation the agent detects if the version of systemd running already knows this option and configures itself accordingly.

If you are not yet convinced by systemd, there is however a simple alternative: Even in systemd-based systems it is (still) possible to use xinetd. This can be run as a service under systemd. To this end install the xinetd package, and then reinstall the Checkmk agent — this should then find and preferentially configure itself for xinetd.

7.3. Invoking over SSH

The best security measure when invoking a Checkmk agent is offered by invoking it via Secure Shell — in Linux in the form of an implementation of OpenSSH. This method is advisable for:

  • The monitoring of Linux servers that are only accessible over the internet.

  • The monitoring of computers in a DMZ.

  • Similar situations in which a TCP-connection from the Checkmk server to the agent is at all possible.

The setting-up is performed in the following steps:

  1. Create an SSH key pair especially for this purpose.

  2. Enable an access to the agent on the target system using these keys.

  3. Disconnect the access over xinetd.

  4. Configure the Checkmk server to use SSH instead of the TCP connection on port 6556.

And now the above procedure, step-by-step with all necessary details:

Creating SSH key pairs

SSH works with a ‘Public-Key-Authorization’. For this purpose first create a pair of complementary keys — one of which is public, and the other private. You may choose between the algorithm rsa, ecdsa or ed25519. In the following example you use ssh-keygen -t ed25519 as the command performed by a site user:

OMD[mysite]:~$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/omd/sites/mysite/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /omd/sites/mysite/.ssh/id_ed25519.
Your public key has been saved in /omd/sites/mysite/.ssh/
The key fingerprint is:
cc:87:34:d2:ed:87:ed:f7:1b:ec:58:1f:7c:23:00:e2 mysite@mycmkserver
The key's randomart image is:
--[ED25519  256--
|                 |
|       . .       |
|      ....      |
|      .=..o     |
|       ES .o    |
|         . o. o  |
|            ...B.|
|             .=.*|
|             . o|

Important: do not enter a passphrase here! Encrypting the file with the secret key will not achieve anything, because you won’t want to have to enter the passphrase every time you start the Checkmk server…​

This will result in two files in the .ssh directory:

OMD[mysite]:~$ ll .ssh
total 8
-rw------- 1 mysite mysite 1679 Feb 20 14:18 id_ed25519
-rw-r--r-- 1 mysite mysite  398 Feb 20 14:18

The private key is called id_ed25519, and is only readable for the site user (-rw-------) and that is what we want! The public key will look something like this:

OMD[mysite]:~$ cat .ssh/
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb6AaqRPlbEmDnBkeIW3Q6Emb5lr2QEbWEQLmA5pb48 mysite@mycmkserver

Enabling access over SSH

The next step must be carried out on every Linux server that is to be monitored over SSH. Log in to the server as root, and — if it does not already exist — create the subdirectory .ssh in the (/root) home directory. With the following command you create the direcotry and assign the correct access rights (700) in one go:

root@linux# mkdir -m 700 /root/.ssh

Now open the authorized_keys file with the (console-based) text editor of your choice. If the file does not already exist, it will be generated automatically by the editor:

root@linux# vim /root/.ssh/authorized_keys

Now copy the content of the public keys into this file. This can be done, for example, with the mouse and copy & paste. Be precise! Every blank character is valid. Be careful to never have two blank characters together, — AND, all of this is a single line! If the file already exists, simply add this as a new line at the end of the file.

Restricting access to agent execution

Now for something very important! The SSH key should be used exclusively for executing the agent. SSH offers such a function under the name Command restriction. For this, set the text command="/usr/bin/check_mk_agent" at the beginning of the line which has just been created — separated from the rest with one blank character. The result will look like this:

command="/usr/bin/check_mk_agent" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb6AaqRPlbEmDnBkeIW3Q6Emb5lr2QEbWEQLmA5pb48 mysite@mycmkserver

Save the changed file, and check the permissions — they must be set to 600:

root@linux# chmod 600 /root/.ssh/authorized_keys
root@linux# ll /root/.ssh/authorized_keys
-rw------- 1 root root 1304 Feb 20 14:36 authorized_keys

Now test access to the agent via SSH once. The first time you will need to confirm the fingerprint of the key by entering "yes". Only after this confirmation will Checkmk be able to access the agent in the background:

OMD[mysite]:~$ ssh root@myhost123
The authenticity of host 'localhost (' can't be established.
ECDSA key fingerprint is 55:34:f9:dd:2b:db:a7:fc:5d:4c:9d:37:28:f7:69:62.
Are you sure you want to continue connecting (yes/no)? yes
Version: 2.0.0p42
AgentOS: linux
Hostname: myhost123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local

By the way, the request for the key fingerprint is received only with the initial use. Should this not function, check the following:

  • Is the SSH server actually installed on the target system?

  • Do the specified files and directories have the correct permissions?

  • Has the syntax in authorized_keys been coded correctly?

  • Has the correct public key been entered?

  • Have you logged in as the correct user (root@…​)?

  • Have you thought of the command="…​."?

For very old target systems, it is also possible that keys with elliptic curves (ed25519 and ecdsa) are not known. In this case you have to create an RSA key as well and append it in the file authorized_keys. SSH will then automatically use the strongest known key for the connection.

Disconnecting access over xinetd

The implementation of SSH is not much use if the access over Port 6556 remains possible as before. To close the port, set Checkmk’s xinetd-service to disabled. Do not delete the complete configuration file, as this will then reappear with the next agent update anyway!

Disabling is performed in /etc/xinetd.d/check-mk-agent:

service check_mk
        type           = UNLISTED
        port           = 6556
        socket_type    = stream
        protocol       = tcp
        wait           = no
        user           = root
        server         = /usr/bin/check_mk_agent
        disable        = yes

Following the restart of xinetd, do not forget:

root@linux# /etc/init.d/xinetd restart

A deinstallation of xinetd is of course also possible — but with the next update of the Checkmk agent it will reactivate itself in any case over Systemd!

Under no circumstances forget to test the setup at the end of this procedure. A connection to Port 6556 should no longer be possible:

OMD[mysite]:~$ telnet myhost123 6556
telnet: Unable to connect to remote host: Connection refused

Migrating Checkmk’s access to SSH

The target system is now ready. Next you have to take care of the configuration of Checkmk itself. You will find the suitable rule set via Setup > Agents > Other integrations> Custom integrations > Individual program call instead of agent access. Create a rule for the affected hosts, and enter the command ssh -T root@$HOSTADDRESS$:

agent linux rule ssh key

After saving and an Activate changes everything should function as intended! For diagnostic purposes the commands cmk -D and cmk -d are well suited, as explained in the article on the command line.

Multiple SSH keys

It is also possible to work with more than one SSH key. Save the keys to any chosen directory. In your rule Individual program call instead of agent access you will need to add the option -i and point it to the appropriate key. Ideally, use $OMD_ROOT as the substitute for the path to the site directory (/omd/sites/mysite). The complete command could be ssh -i $OMD_ROOT/.ssh/my_key -T root@$HOSTADDRESS$ and like this the configuration will also be runable in a site with a different name:

agent linux rule multiple ssh keys

Different host groups can use different SSH keys, if multiple different rules are used.

Typical error messages when dealing with SSH

When you setup agent access via SSH you might encounter error messages in the summary of the critical service Check_MK, like Agent exited with code 255.

Information on how to resolve such errors can be found in the dedicated Knowledge base article Executing the Linux agent over ssh.

7.4. Inbuilt encryption

The Checkmk agent can encrypt its own data without additional tools. Strictly speaking, this is no substitute for an access control. But since an attacker cannot send commands, and can’t work with encrypted output data, as a solution this is nearly as effective.

The complexity with the use of encryption and the additional CPU-load incurred are both less than with the SSH method as described above, the method which we nevertheless still recommend when transferring data over the internet.

The encryption of course requires a suitable configuration, both on the server and in the agents. This can be created manually (Raw Edition) or with the Agent Bakery (Enterprise Editions).

Installation without the Agent Bakery

Even without the Agent Bakery the first step is still performed in Setup by creating a Setup > Agents > Access to Agents > Checkmk agent > Encryption rule in the rule set. The rule should apply to all hosts for which an encryption will be implemented. SNMP-hosts ignore this setting, so it is not necessary to explicitly exclude them.

agent linux encrypt

The Encryption for agent setting is important. As long as the rule retains the default Disable, everything naturally remains as in the old setup. The choice is thus between:

  • Enable: Encryption is active, but data from unencrypted agents will continue to be accepted.

  • Enforce: Encryption is active, and only encrypted data will be accepted.

It makes sense to begin with Enable initially. Once you are satisfied that all agents have been converted to encryption, switch over to Enforce, in order to identify hosts which still send data as clear text.

The encryption makes use of a shared password which must be stored on both the Checkmk server and on the agent as a simple text (a ‘Shared secret’), and which will be entered here. Choose a random password and have it ready for the second step — the configuration of the agent.

Create the file /etc/check_mk/encryption.cfg with the following contents on the agent:


Of course use your own password in PASSPHRASE. This file should definitely be secured against reading by other users:

root@linux# chmod 600 /etc/check_mk/encryption.cfg

Now the following tests can be made (for information on these, see the article on Checkmk’s command line):

  • Running check_mk_agent on the target system should produce a nonsense jumble of characters.

  • Accessing the agent via telnet myhost123 6556 from the Checkmk server should produce the same jumble.

  • The command cmk -d myhost123 on the Checkmk server should show the normal clear text.

Implementing with the Agent Bakery

CEE The implementation of the encryption with the Agent Bakery is very simple. By creating the rules as just described, one is basically finished. Now only new agents need to be baked and distributed. The file /etc/check_mk/encryption.cfg will be automatically generated and added to the agent package.

8. Monitoring Linux over SNMP

Since an easily-installed SNMP agent is available for Linux, the question arises whether it is possible or even desirable to monitor Linux over SNMP? The answer is quite simple: possible yes, sensible no. Why is this so?

  • The monitoring data from the SNMP agents is very limited, so that in any case Checkmk agents are required for a halfway adequate monitoring.

  • The SNMP agent delivers no useful data that isn’t already provided by the Checkmk agent.

  • The SNMP agent is more time-consuming to install.

  • Last but not least, the SNMP protocol consumes far more CPU and network resources than a normal Checkmk monitoring.

There are however a few situations in which monitoring per SNMP in addition to the normal agents can be sensible. Such a case would be when either a user’s software, or a hardware monitoring tool supplied by the server’s manufacturer only delivers data via SNMP.

In such a case, set in the Setup menu, Monitoring agents box the Checkmk agent / API integrations option to a value with Checkmk agent (Configured API integrations and Checkmk agent or API integrations if configured, else Checkmk agent) and the SNMP option to SNMP v2 or v3 or SNMP v1. Services that are accessible over SNMP as well as per Checkmk agent (e.g. CPU-load, file systems, network cards) will be automatically retrieved by Checkmk agent and not over SNMP. In this way a duplicated data transfer will be automatically prevented.

9. Hardware monitoring

9.1. Fundamentals

To be as comprehensive as possible the monitoring of a Linux server must of course include the hardware. This is achieved in part directly with the Checkmk agents, and partly also using special plug-ins. Additionally, there are also cases in which monitoring can be implemented per SNMP, or even over a separate management board.

9.2. Monitoring the SMART values

Modern hard drives almost always use S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology). This system continuously logs data on the condition of the HDD or SSD, and with the smart plug-in Checkmk can retrieve these values and evaluate the most important of them. In order that the plug-in functions following an installation, the following preconditions must be met:

  • The smartmontools packet must be installed. This can be installed on all modern distributions with the appropriate packet manager.

  • If the drives are connected via a RAID-controller and this allows access to the SMART-values, the appropriate tool must also be installed. tw_cli (3ware) and megacli (LSI) are supported.

If these prerequisites have been satisfied, and the plug-in has been installed, the data will be automatically read and appended to the agent’s output. In Checkmk the new services can then also be activated directly:

agent linux smart stats

If — as shown in the screenshot — a cmd_timeout occasionally occurs, set the smart plug-in to asynchronous execution at intervals of a few minutes.

9.3. Monitoring with the help of IPMI

IPMI (Intelligent Platform Management Interface) is an interface for hardware management which, among other functions, enables the monitoring of hardware. Checkmk uses freeipmi for this, in order to access the hardware directly and without a network. It is installed from the package source and is ready for immediate use, so that the data will be transmitted at the next polling by Checkmk.

If freeipmi is not available, or there are other grounds preventing an installation, ipmitool can also be used. This is often already present on a system and must only served by an IPMI hardware driver — which can be provided by the openipmi package, for example. Likewise, nothing more needs to be done here subsequently. The data will be recorded by Checkmk automatically.

For error diagnosis the tools can also be executed manually in a host shell. If the freeipmi packet has been installed it can be used to control the function:

root@linux# ipmi-sensors Temperature
32 Temperature_Ambient 20.00_C_(1.00/42.00) [OK]
96 Temperature_Systemboard 23.00_C_(1.00/65.00) [OK]
160 Temperature_CPU_1 31.00_C_(1.00/90.00) [OK]
224 Temperature_CPU_2 NA(1.00/78.00) [Unknown]
288 Temperature_DIMM-1A 54.00_C_(NA/115.00) [OK]
352 Temperature_DIMM-1B 56.00_C_(NA/115.00) [OK]
416 Temperature_DIMM-2A NA(NA/115.00) [Unknown]
480 Temperature_DIMM-2B NA(NA/115.00) [Unknown]

Once ipmitool has been installed the data output can be checked with the following command:

root@linux# ipmitool sensor list
UID_Light 0.000 unspecified ok na na 0.000 na na na
Int._Health_LED 0.000 unspecified ok na na 0.000 na na na
Ext._Health_LED 0.000 unspecified ok na na 0.000 na na na
Power_Supply_1 0.000 unspecified nc na na 0.000 na na na
Fan_Block_1 34.888 unspecified nc na na 75.264 na na na
Fan_Block_2 29.792 unspecified nc na na 75.264 na na na
Temp_1 39.000 degrees_C ok na na -64.000 na na na
Temp_2 16.000 degrees_C ok na na -64.000 na na na
Power_Meter 180.000 Watts cr na na 384.00

9.4. Manufacturer-specific tools

Many server producers offer their own tools for recording hardware information and delivering this via SNMP. The following prerequisites apply to be able to retrieve this data and provide it to Checkmk:

  • An SNMP server has been installed on the Linux host.

  • The manufacturer’s tool has been installed — e.g., Dell’s OpenManage or Supermicros SuperDoctor.

  • The host has been configured in Checkmk for an additional monitoring per SNMP.

The new services for hardware monitoring thereby supported will then be automatically recognised. No further plug-ins will be required.

9.5. Supplementary monitoring over the management board

A management board can be configured for every host and additional data collected per SNMP. The services thereby recognised will likewise be assigned to the host.

Setting up a management board is very easy. In the host’s attributes, just enter the protocol, the IP-address and the access data for SNMP, and save the new settings:

agent linux snmp management board

With a service discovery the newly-discovered services will then be activated as usual.

10. Files and directories

10.1. Paths on the monitored host

File pathFunction


Installation location of the Checkmk agent on the target system.


The home directory for agent extensions


Plug-ins which extend the agent for collecting supplementary monitoring data. Plug-ins can be written in any available programming language.


The user’s own local checks.


Data location for the Checkmk agent on the target system.


Location of cached data from particular sections. These will be attached to the agent as long as the cache data is valid.


Location of job data. These files will be added to the output on every agent execution.


Contains data created by e.g., cron jobs, and that includes its own section. These are also appended to the agent output.


Location of configuration files for the agents.


Configuration file for MRPE — for the execution of standard, Nagios-compatible check plug-ins.


Configuration for the encryption of the agent data.


Configuration file for disabling specific sections of the agent


Configuration for the xinetd, which connects the agent’s output to TCP-Port 6556.

10.2. Paths on the Checkmk server



The base directory for own files which are to be delivered with baked agents.


Example configuration file for disabling sections

On this page