1. Introduction

Linux systems can be especially well monitored by Checkmk. This has less to do with the Checkmk development team ‘feeling at home’ with Linux — rather it has more to do with Linux being a very open system, and it providing numerous very well documented and easily queried interfaces which enable a comprehensive monitoring operation.
The installation of a monitoring agent is essential, since most of the interfaces are not in themselves accessible over a network. For this reason Checkmk has its own agent for the monitoring of Linux. This is easy to install and is economical in its use of system resources.
The Checkmk agent for Linux is…
… minimalistic, since it requires little CPU, RAM and hard drive space.
… transparent, because it is a shell script, in which you can see which command it is executing.
… secure, since it permits no access from the network whatsoever.
The agent consists of a simple shell script that is installed in
/usr/bin/check_mk_agent
, and which invokes available system commands
sequentially in order to seek data for the monitoring. It routes its output
using either xinetd
or systemd
to TCP-Port 6556, or it is
alternatively called per SSH. Should you like neither of these procedures
you can also implement your own methods for Checkmk to receive data from agents.
Monitoring of information that the standard agent doesn’t provide can be added in the form of agent plug-ins. In the Enterprise Editions you can bundle all settings and plug-ins together with the agent into an RPM or DEB packet using the Agent Bakery, this packet can then be installed with a single command and/or even be updated fully automatically.
2. Installation
2.1. Various options
Checkmk provides a number of ways to install a Linux agent — from the manual installation of its separate components, to a fully automated deployment. Some of these are only available in the Enterprise Editions:
Method | Description | CRE | CEE |
---|---|---|---|
Included as standard RPM/DEB-Packet | Simple installation of a standard agent with a manual configuration over configuration files | X | X |
RPM/DEB-Packet from the Agent Bakery | Configuration via the GUI — individual configuration of each host is possible | X | |
The packet from the agent bakery is initially installed manually or by using a script, and subsequently from there updated automatically. | X | ||
Manual installation | The individual files, without a packet, are copied to the target system and then | X | X |
2.2. Installation using a RPM/DEB-Packet
The agent files are located in the
Monitoring agents WATO module. In the Enterprise Editions this will lead to the
Agent Bakery. From here, with the Agent files button
the list of the agent files can be opened:

Everything required is in the first section called Packaged Agents.
Finished RPM and DEB packets for the installation of a Linux agent with
standard settings can be found here (the package for Windows
with the .msi
extension is also here):

Whether you require RPM or DEB depends on the Linux distribution on which the package is to be installed:
Packet | File extension | For installation on |
---|---|---|
RPM | .rpm | Red Hat Enterprise Linux, Fedora, CentOS, openSUSE, SLES, and derivatives of these |
DEB | .deb | Debian, Ubuntu, and all other DEB-based distributions |
Attention: in the Linux/Unix-Agents section can be found the straight agent
scripts for the manual installation to /usr/bin/
.
Only use these if the packaged agents do not function on your system, or if you
wish to monitor Unix derivatives other than Linux.
Exotic Linux distributions
Also, if the Checkmk-server only supports Enterprise-Linux distributions that are still supported by their producer, the Checkmk-agent is in such a case much less demanding. It supports every Linux distribution – even ancient ‘dinosaurs’ on which a Version 2.4 core is running! It is however possible that not all plug-ins will run correctly in the agent, but the basic monitoring itself will function.
Installation with RPM
The installation of an agent with the RPM package is very simple:
Download the agent packet with the
*.rpm
file extension.Copy it to the target system to be monitored with
scp
, WinSCP or similar.Install it on that system as
root
using therpm -U
command.Next install the
xinetd
packet from your distribution, if it is not already installed.
Here is an example of a package installation:
root@linux# rpm -U check-mk-agent-1.6.0p11-1.noarch.rpm
By the way, the -U
stands for ‘Update’, but it can also perform an
initial installation correctly. This means that the same command can also be
used later for an update to a newer version.
The subsequent installation of xinetd
on Red Hat/CentOS is performed with:
root@linux# yum install xinetd
For SUSE Linux use zypper
:
root@linux# zypper install xinetd
Following the installation an agent is immediately active and can be queried through TCP Port 6556.
Installation with DEB
This procedure is the same as for RPM, only the commands are different. An installation and a later update of the agent are both performed with:
root@linux# dpkg -i check-mk-agent_1.6.0p11-1_all.deb
A subsequent installation of xinetd
:
root@linux# apt-get install xinetd
Get package via HTTP
Sometimes scp
or WinSCP can be very laborious. The packet can also be
loaded directly from the Checkmk-server to the target system over HTTP.
For this purpose the agent file downloads have deliberately been made possible
without needing a login. The files after all contain no secrets. Anyone can download
and install Checkmk themselves and then access the files.
The easiest method is to use wget
. The URLs can be identified via the browser.
If you know the package’s name you can simply assemble the URL name yourself.
Then simply set /mysite/check_mk/agents/
before the file name:
RPM actually has an inbuilt wget
— with this you can perform the download and
the installation in a single action:
2.3. Installation with the Agent-Bakery
The
Checkmk Enterprise Editions utilises the Agent-Bakery via a
WATO-Modul to package individually-customised agents. This process will be
described in the general chapter covering the Agents.
Installation of the baked packets is performed exactly as described above.
2.4. Automatic updates
If you use the agent bakery, the agent’s automatic updates can set this up.
This process is described in its own article.
2.5. Manual installation
The manual installation of an agent is rarely necessary, but is not very difficult. As well as the agent files from the site, you also require the Linux/Unix agents selection menu — here you will find the {CMK} Agent for Linux file:

Load this file to the target system, and copy it to a directory that is
executable for root
. /usr/local/bin/
is very well suited
for this, since it is found in the search path and has been conceived for the
user’s own extensions. Here you can also work directly with wget
:
root@linux# cd /usr/local/bin
root@linux# wget http://mycmkserver/mysite/check_mk/agents/check_mk_agent.linux
root@linux# mv check_mk_agent.linux check_mk_agent
root@linux# chmod 755 check_mk_agent
Please don’t forget the last two commands — these remove the .linux
file extension and make the file executable. If everything has been done
correctly the agent should now simply be an executable command which will generate its usual output.
This also works if you are not in /usr/local/bin
.
The | head
truncates everything after the 11th line:
root@linux# check_mk_agent | head
<<<check_mk>>>
Version: 1.6.0p11
AgentOS: linux
Hostname: mycmkserver
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>
In the case of a very old distribution which does not recognise the timeout
command, load the small program waitmax
from the agent page and install
it likewise in /usr/local/bin
. Both timeout
and waitmax
perform
the same function — they force a timeout when executing a program:
root@linux# timeout --help
Usage: timeout [OPTION] DURATION COMMAND [ARG]...
or: timeout [OPTION]
Start COMMAND, and kill it if still running after DURATION.
Waitmax was developed as a Checkmk component at a time when timeout
was not widely used. It has almost the same call syntax:
root@linux# waitmax --help
age: waitmax [-s SIGNUM] MAXTIME PROGRAM [ARGS...]
Execute PROGRAM as a subprocess. If PROGRAM does not exit before MAXTIME
seconds, it will be killed with SIGTERM or an alternative signal.
-s, --signal SIGNUM kill with SIGNUM on timeout
-h, --help this help
-V, --version show version an exit
Should you wish to configure or extend the agent, you will need to create
the required directories yourself. The location for the three required
directories is hard coded in variables that begin with MK_
,
and which will also be provided to the plug-ins over the environment:
root@linux# grep 'export MK_' check_mk_agent
export MK_LIBDIR="/usr/lib/check_mk_agent"
export MK_CONFDIR="/etc/check_mk"
export MK_VARDIR="/var/lib/check_mk_agent"
These three directories should be created (with the standard permissions 755):
root@linux# mkdir /usr/lib/check_mk_agent /etc/check_mk /var/lib/check_mk_agent
Should you want to change the file path, simply edit it in
/usr/local/bin/check_mk_agent
.
If you fundamentally want to call up the agents over SSH, a configuration for
the xinetd
is required, you just additionally need the SSH-configuration.
How that is achieved will be described below.
The configuration per xinetd
enables an access of the agent data
via TCP Port 6556, and is the standard method in local networks.
Install the xinetd
package for this, and create the following file:
service check_mk
{
type = UNLISTED
port = 6556
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/local/bin/check_mk_agent
only_from = 10.118.14.5 10.118.14.37
disable = no
}
Here enter the IP-Address of your Checkmk-server that will be permitted to access
the agent under only_from
.
Then only an activation is required and the agent will be ready:
root@linux# /etc/init.d/xinetd reload
If you’re using systemd, you should execute the following command instead:
root@linux# systemctl reload xinetd.service
3. Testing, and error diagnosis
3.1. Overview
Once the agent has been installed, the obvious question will surely be how to test whether everything has been done correctly. All of the possibilities that are available from the Checkmk-server are described in the general chapter on the agents. There are of course further diagnostic possibilities when one is directly logged into the target system itself.
Since the ‘agent’ is basically nothing more than a simple program that obtains data from your system and outputs it as loosely-formated text, you can also invoke it as a program, and in fact do so quite easily:
root@linux# check_mk_agent
<<<check_mk>>>
Version: 1.6.0p11
AgentOS: linux
Hostname: myhost123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>
udev devtmpfs 8155492 4 8155488 1% /dev
tmpfs tmpfs 1634036 1204 1632832 1% /run
/dev/sda5 ext4 226298268 176973752 37806104 83% /
none tmpfs 4 0 4 0% /sys/fs/cgroup
Because the output can be rather long, less
can also be very practical
here (you can quit it with the 'Q'-key):
root@linux# check_mk_agent | less
This output does not of course prove whether the agent is also accessible over the network. But in this way it can be tested whether all of the desired data is present in the output.
Incidentally — it is not essential to be root
to be able to invoke
the agent, however it is possible that the output could be missing some information
which requires root
-permissions to obtain
(e.g., multipath information and the output from ethtool
).
3.2. Debug mode
So that possible erroneus outputs from defective plug-ins or commands do not
‘corrupt’ the real production data, as a matter of principle the agent suppresses
the standard error channel. If one is looking for a specific problem this can be
reactivated by invoking the agent with a special debug mode. This is done
with the -d
option.
In so doing all of the shell commands which the agent executes will be output.
So that you can work with less
here, the standard output and error
channels must be combined with 2>&1
:
root@linux# check_mk_agent -d 2>&1 | less
4. Integration of classic check plug-ins
4.1. Executing plug-ins over MRPE
If you have migrated your monitoring from a Nagios-based solution to Checkmk, it cannot be ruled out that you have existing check plug-ins of a standard form for which no counterpart in Checkmk (yet) exists. In most cases these will be self-written plug-ins in Perl or Shell.
The Checkmk-agent offers a simple mechanism that enables such plug-ins still to be used: MK’s Remote Plugin Executor or MRPE for short. The name is intentionally an analogy to NRPE in Nagios, for which it performs the same function.
The MRPE is integral in agents and is configured with a simple text file
which you yourself can save in /etc/check_mk/mrpe.cfg
.
In this file enter one plug-in call per line — together with the name that
Checkmk should use for the service that is to be automatically created.
Here is an example:
Foo_Application /usr/local/bin/check_foo -w 60 -c 80
Bar_Extender /usr/local/bin/check_bar -s -X -w 4:5
If the agent is allowed to run locally, for each plug-in a new section will be
found with the title <[mrpe]>
, containing the name,
exit code and output from the the plug-in. This can be verified with the
following practical grep
-command:
root@linux# check_mk_agent | grep -A1 '^...mrpe'
<<<mrpe>>>
(check_foo) Foo_Application 0 OK - Foo server up and running
<<<mrpe>>>
(check_bar) Bar_Extender 1 WARN - Bar extender overload 6.012|bar_load=6.012
The 0
or 2
in the output stand for the plug-in’s exit codes
and conform to the standard pattern: 0
= OK, 1
=
WARN, 2
= CRIT and 3
= UNKNOWN.
The rest will be taken care of by Checkmk automatically. Once a service discovery has been performed for the host, the two new services will be shown as available. It will look like this:

Incidentally: due to the file’s syntax the name is not permitted to contain blank characters.
With the help of the same syntax as used in URLs, a space can be replaced by
%20
(the ASCII-Code 32 for ‘space’ is Hexadecimal 20):
Foo[hilite]%20#Application /usr/local/bin/check_foo -w 60 -c 80
Bar[hilite]%20#Extender /usr/local/bin/check_bar -s -X -w 4:5
4.2. Asynchronous execution
Please note that all plug-ins running in mrpe.cfg
will be executed
synchronously and sequentially.
The plug-ins should thus not have overly long runtimes. If a plug-in hangs,
all following plug-ins will be delayed. This could lead to the agent’s complete
retrievals under Checkmk entering a timeout, meaning that the host can no longer
be reliably monitored.
If you really require longer running plug-ins, these should be converted to
asynchronous processing, thus avoiding the problem described above.
For this, define a time frame in seconds for which a calculated result will be
valid — for example, 300
for five minutes.
To achieve this, set the expression (interval=300)
following the service
name in mrpe.cfg
:
Foo_Application (interval=300) /usr/local/bin/check_foo -w 60 -c 80
Bar_Extender /usr/local/bin/check_bar -s -X -w 4:5
This will have a number of effects:
The plug-in will be executed in a background process and will no longer hinder the agent’s processing.
Because the agent no longer waits for the execution, the result will be delivered until the next call of the agent.
The plug-in will not be run before the defined 300 seconds have elapsed. Until that time the old result will be recycled.
With this method tests that require very long processing times, as well as longer intervals, can also be run without needing to make changes to the configuration on the Checkmk-server.
4.3. MRPE with the Agent Bakery
Proud owners of the Enterprise Editions can also configure MRPE with the
Agent Bakery. The rule set
Monitoring Agents ⇒ Rules ⇒ Generic Options ⇒ Execute MRPE Checks is responsible for this.
There you can configure the same things as described above.
The appropriate
mrpe.cfg
file will then be automatically generated by the bakery.

Baking the plug-ins
Likewise the Check plug-ins themselves can be delivered with the package. With these the agent will be complete and will require no manual installation of additional files. This is all performed as follows:
Create the directory
local/share/check_mk/agents/custom
on the Checkmk-serverCreate a subdirectory, e.g.
my_mrpe_plugins
in the above directoryThen create the subdirectory
bin
in thisCopy your plug-ins to this directory
Create a rule in Monitoring Agents ⇒ Rules ⇒ Generic Options ⇒ Deploy custom files with agent.
Select
my_mrpe_plugins
, save and bake it!
The Check plug-ins will be installed in your agent’s standard-bin
-directory.
By default this is /usr/bin
. Then when configuring the MRPE-checks,
/usr/bin/check_foo
will be required instead of /usr/local/bin/check_foo
.
5. Extending agents with plug-ins
5.1. What are plug-ins?
The standard agent /usr/bin/check_mk_agent
contains a whole series of
sections which provide monitoring data for various checks which will then be
found automatically by a service discovery. These include all of an operating
system’s important monitorings.
Additionally, there is also the possibility of augmenting an agent with plug-ins. These are small scripts or programs which are invoked by an agent to include further sections with additional monitoring data. The Checkmk-Project delivers a whole series of such plug-ins, which — when correctly installed and configured — via a service discovery can provide new checks automatically.
Why are these plug-ins not simply built into the standard agent? For each plug-in one of the following reasons prevents such an integration:
The plug-in is coded in a programming language other than Shell and thus cannot implemented inline (example:
mk_logwatch
).The plug-in in any case requires a configuration without which it will not function (example:
mk_oracle
).The plug-in is so specialised that most users will not require it (example:
plesk_domains
).
5.2. Manual installation of plug-ins
The plug-ins for Linux and UNIX included with the project are all located on the
Checkmk-server in share/check_mk/agents/plugins
.
Additionally, these are available via the download page for the agents in WATO (as
described at the beginning of this article) in the Linux/Unix Agents - Plugins menu box:

For all of our standard agent plug-ins there are also the matching check plug-ins which can evaluate the data and create services from these. They are ready to use and do not require an extra installation.
Before installing a plug-in in an agent, please have a look at its associated file. Important tips for the correct use of the plug-in can often be found there.
The actual installation is then simple: simply save the file to
/usr/lib/check_mk_agent/plugins
.
When doing this ensure that the file is executable.
If not, use a chmod 755
, otherwise the agent will not run the plug-in.
Especially if the files are not transferred over scp
, rather via HTTP
from the download page, the execution permissions will be lost!
Once a plug-in is executable and in the correct directory, it will be invoked
by the agent and a new section will be generated in the agent’s output.
This section usually has the same name as the plug-in. Complex plug-ins (e.g.,
mk_oracle
) in fact create a whole series of sections.
In older versions of the Checkmk agent the plug-in directory can be in another location. If case of uncertainty, or you don’t know if this applies in your setup, the directory can be identified with the following:
root@linux# grep MK_LIBDIR= /usr/bin/check_mk_agent
export MK_LIBDIR="/usr/lib/check_mk_agent"
5.3. Configuration of plug-ins
Some plug-ins require a configuration file in /etc/check_mk/
to be able
to function. With others a configuration is optional and allows special features or
customisation. Others simply work as is. There are various sources of relevant information:
The documentation for the matching check plug-ins in the Check plugins WATO-module
Comments in the plug-in itself (often very helpful!)
A relevant article in this handbuch (e.g., on the monitoring of Oracle)
5.4. Asynchronous execution
Plug-ins can process asynchronously in the same way as with MRPE. This is very useful if the plug-ins have a very long runtime, and the acquired status data in any case does not need to be refreshed every minute.
An asynchronous execution is not configured with a file. Instead, create a subdirectory
in plugins
whose name is a numeric, representing a count in seconds. Plug-ins in this
directory will not only be executed asynchronously, at the same time by specifying the second
count a minimum waiting time is also specified before the plug-in can be run again.
If the agent is again called before the specified time has elapsed, it will use
cached data from the last run of the plug-in. In this way an interval longer than
the typical one minute can in effect be configured for the plug-in.
The following example shows how by specifying a 5 minute interval the >my_foo_plugin` plug-in can be changed from synchronous to asynchronous execution:
root@linux# cd /usr/lib/check_mk_agent/plugins
root@linux# mkdir 300
root@linux# mv my_foo_plugin 300
Please note that a few plug-ins are set up internally to execute
asynchronously by default. Among these is mk_oracle
.
Always install such plug-ins directly in /usr/lib/check_mk_agent/plugins
!
5.5. Installing plug-ins using the bakery
Plug-ins included with Checkmk can be configured using the Agent Bakery. This not only takes care of the installation of the plug-ins themselves, but also for the correct generation of the configuration file should one be required.
Each plug-in is configured with an agent rule. The appropriate rule set can be found in Monitoring agentes ⇒ Agent plugins:

5.6. Manual execution of plug-ins
Since agent plug-ins are executable programs, for testing and diagnosis they can be manually started. There are however plug-ins which require specific environment variables to be set by agents — so that they can find their configuration file, for example. Set these variables manually before execution:
root@linux# export MK_LIBDIR=/usr/lib/check_mk_agent
root@linux# export MK_CONFDIR=/etc/check_mk
root@linux# export MK_VARDIR=/var/lib/check_mk_agent
root@linux# /usr/lib/check_mk_agent/plugins/mk_foobar
<<<foobar>>>
FOO BAR BLA BLUBB 17 47 11
Some plug-ins have special invocation options for debugging. Just have a look in the plug-in!
6. Security
6.1. Considerations
Nowadays everything must be safe — and of course monitoring is no exception. Because the monitoring agent is installed on every server being monitored, a security problem here can have serious consequences.
For this reason emphasis is placed on security in the design of Checkmk, and from the earliest days of Checkmk security has been an unshakeable principle: An agent reads no data from the network — full stop. Consequently it is quite impossible that an attacker could sneak any type of command or script element over the monitoring port 6556.
This alone provides such a high level of security that most users do without additional measures in the LAN. If the system being monitored is only accessible over an insecure internet connection, then of course quite different precedures are necessary, and an encryption with SSH would certainly be the ideal first choice.
The Checkmk agent additionally includes an inbuilt encryption, which represents a good compromise between security and complexity. In the following section we will show all of the options for protection in detail.
6.2. Restricting of access over IP-addresses
Even if an attacker is unable to execute commands, the monitoring data could be useful to him as among other info, the data includes a list of all processes running on the system. Is is therefore best that the data is accessible to nobody.
xinetd
If the Checkmk-agents are as usual enabled over the xinetd
, it is very
simple and effective to restrict the access to specific IP-addresses — to those
of the monitoring server of course. This is easily done, and was in fact
to be seen in the example above:
service check_mk
{
type = UNLISTED
port = 6556
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/bin/check_mk_agent
only_from = 10.118.14.5 10.118.14.37
disable = no
}
Agent Bakery users can configure the permitted IP-addresses using the WATO rule set Monitoring agents ⇒ Rules ⇒ Generic options ⇒ Allowed agent access via IP address.
An attacker can of course easily falsify their IP-address and thus make a connection to an agent — but it is very unlikely that it will receive an answer since this goes to the genuine monitoring server. Or it really does receive an answer, but the Checkmk-server looks down the pipe and will very quickly record an error.
Systemd
Since Systemd is now the new toy that everyone uses, Linux distributors are working hard to retire the good old xinetd. The prepackaged Linux-Agent (not one from the Bakery!) already installs itself with Systemd instruments, if the target system is based on Systemd and no xinetd is available.
Systemd however cannot perform such simple tasks as only_from
.
One will be bluntly referred to iptables
.
Should your agent thus, without xinetd, be invoked purely with Systemd,
there is unfortunately no simpler possibility for restricting the IP-addresses,
other than using a firewall configuration.
If you are not yet one of the Systemd evangelists, there is however a simple
alternative: Even in Systemd-based systems it is (still) possible to use xinetd.
This can be run as a service under Systemd — and again the method with
only_from
as described above also works. To this end install
the xinetd
packet, and then reinstall the Checkmk-agent — this
should then find and preferentially install xinetd.
6.3. Invoking over SSH
The ultimate security for invoking a Checkmk-agent is offered by invoking it via Secure Shell — in Linux in the form of an implementation of OpenSSH. This method is advisable for:
The monitoring of Linux servers that are only accessible over the internet.
The monitoring of computers in a DMZ.
Similar situations in which a TCP-connection from the Checkmk-Server to the agent is at all possible.
The setting-up is performed in the following steps:
Create an SSH-key pair especially for this purpose.
Enable an access to the agent on the target system using these keys.
Disconnect the access over xinetd.
Configure the Checkmk server to use SSH instead of the TCP connection on port 6556.
And now the above procedure, step-by-step with all necessary details:
Creating SSH-key pairs
SSH works with a ‘Public-Key-Authorization’. For this purpose first create
a pair of complementary keys — one of which is public, and the other private.
This is performed by an instance user with ssh-keygen -t ed25519
:
OMD[mysite]:~$ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/omd/sites/mysite/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /omd/sites/mysite/.ssh/id_ed25519.
Your public key has been saved in /omd/sites/mysite/.ssh/id_ed25519.pub.
The key fingerprint is:
cc:87:34:d2:ed:87:ed:f7:1b:ec:58:1f:7c:23:00:e2 mysite@mycmkserver
The key's randomart image is:
--[ED25519 256--
| |
| . . |
| .... |
| .=..o |
| ES .o |
| . o. o |
| ...B.|
| .=.*|
| . o|
-----------------
Important: do not enter a passphrase here! Encrypting the file with the secret key will not achieve anything, because you won’t want to have to enter the passphrase every time you start the Checkmk-server…
This will result in two files in the .ssh
directory:
OMD[mysite]:~$ ll .ssh
total 8
-rw------- 1 mysite mysite 1679 Feb 20 14:18 id_ed25519
-rw-r--r-- 1 mysite mysite 398 Feb 20 14:18 id_ed25519.pub
The private key is called id_ed25519
, and is only readable for the instance
user (-rw-------
) and that is what we want!
The public key id_ed25519.pub
will look something like this:
OMD[mysite]:~$ cat .ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb6AaqRPlbEmDnBkeIW3Q6Emb5lr2QEbWEQLmA5pb48 mysite@mycmkserver
Enabling access over SSH
The next step must be carried out on every Linux server that is to be
monitored over SSH. Log in to the server as root
, and — if it
does not already exist — create the subdirectory .ssh
in
the (/root
) home directory:
root@linux# mkdir /root/.ssh
The permissions for this directory must be 700
, so that it
will be recognised by SSH. If you have created the directory yourself,
you will also require:
root@linux# chmod 700 /root/.ssh
Now open the authorized_keys
file with the (console-based) text
editor of your choice. If the file does not already exist, it will be
generated automatically by the editor:
root@linux# vim /root/.ssh/authorized_keys
Now copy the content of the public keys into this file. This can be done, for example, with the mouse and copy & paste. Be precise! Every blank character is valid. Be careful to never have two blank characters together, — AND, all of this is a single line! If the file already exists, simply add this as a new line at the end of the file.
Restricting access to agent execution
Now for something very important! The SSH-key should be used exclusively
for executing the agent. SSH offers such a function under the name
Command restriction. For this, set the text command="/usr/bin/check_mk_agent"
at the beginning of the line which has just been created — separated from
the rest with one blank character. The result will look like this:
command="/usr/bin/check_mk_agent" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb6AaqRPlbEmDnBkeIW3Q6Emb5lr2QEbWEQLmA5pb48 mysite@mycmkserver
Save the changed file, and check the permissions — they must be set to 600
:
root@linux# chmod 600 /root/.ssh/authorized_keys
root@linux# ll /root/.ssh/authorized_keys
-rw------- 1 root root 1304 Feb 20 14:36 authorized_keys
Now an access via SSH to the agent from the monitoring server should be possible. This can be checked with:
OMD[mysite]:~$ ssh root@myhost123
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 55:34:f9:dd:2b:db:a7:fc:5d:4c:9d:37:28:f7:69:62.
Are you sure you want to continue connecting (yes/no)? yes
<<<check_mk>>>
Version: 1.6.0p3
AgentOS: linux
Hostname: myhost123
AgentDirectory: /etc/check_mk
DataDirectory: /var/lib/check_mk_agent
SpoolDirectory: /var/lib/check_mk_agent/spool
PluginsDirectory: /usr/lib/check_mk_agent/plugins
LocalDirectory: /usr/lib/check_mk_agent/local
<<<df>>>
By the way, the request for the key fingerprint is received only with the initial use. Should this not function, check the following:
Is the SSH-server actually installed on the target system?
Do the specified files and directories have the correct permissions?
Has the syntax in
authorized_keys
been coded correctly?Has the correct public key been entered?
Have you logged in as the correct user (
root@…
)?Have you thought of the
command="…."
?
Disconnecting access over xinetd
The implementation of SSH is not much use if the access over Port 6556
remains possible as before.
To close the port, set Checkmk’s xinetd-service to disabled
.
Do not delete the complete configuration file, as this will then
reappear with the next agent update anyway!
Disabling is performed in /etc/xinetd.d/check-mk-agent
:
service check_mk
{
type = UNLISTED
port = 6556
socket_type = stream
protocol = tcp
wait = no
user = root
server = /usr/bin/check_mk_agent
disable = yes
}
Following the restart of xinetd, do not forget:
root@linux# /etc/init.d/xinetd restart
A deinstallation of xinetd is of course also possible — but with the next update of the Checkmk-agent it will reactivate itself in any case over Systemd!
Under no circumstances forget to test the setup at the end of this precedure. A connection to Port 6556 should no longer be possible:
OMD[mysite]:~$ telnet myhost123 6556
Trying 10.118.15.23...
telnet: Unable to connect to remote host: Connection refused
Migrating Checkmk’s access to SSH
The target system is now ready. Now only the configuration of Checkmk itself
needs to be done. This is performed via the
Datasource programs ⇒ Individual program call instead of agent access rule set.
Create a rule here for the affected hosts, and enter the command ssh -T
-oStrictHostKeyChecking=no root@<IP>
:

After saving and an Activate changes everything should function as intended!
For diagnostic purposes the commands cmk -D
and cmk -d
are well
suited, as explained in the article on the command line.
Details concerning the ‘Datasource programs’ can be viewed in its own article.
Multiple SSH-keys
It is also possible to work with more than one SSH-key. Save the keys to any chosen
directory. With ‘Datasource program’ the data path to the appropriate private key
must include the -i option. Ideally, use $OMD_ROOT
as the substitute
for the path to the instance directory (/omd/sites/mysite
).
The configuration will then also be runable in an instance with a different name:

Different host groups can use different SSH-keys, if multiple different rules are used in Datasource programs.
6.4. Inbuilt encryption
The Checkmk agent can encrypt its own data without additional tools. Strictly speaking, this is no substitute for an access control. But since an attacker cannot send commands, and can’t work with encrypted output data, as a solution this is nearly as effective.
The complexity with the use of encryption and the additional CPU-load incurred are both less than with the SSH method as described above, the method which we nevertheless still recommend when transferring data over the internet.
The encryption of course requires a suitable configuration, both on the server and in the agents. This can be created manually (Raw Edition) or with the Agent Bakery (Enterprise Editions).
Installation without the Bakery
Even without the Agent Bakery the first step is still performed in WATO: by creating a Host & Service Parameters ⇒ Access to agents ⇒ Check_MK Agent ⇒ Encryption rule in the rule set. The rule should apply to all hosts for which an encryption will be implemented. SNMP-hosts ignore this setting, so it is not necessary to explicitly exclude them.

Important is the Encryption for agent setting. As long as the rule retains the default Disable, everything naturally remains as in the old setup. The choice is thus between:
Enable: Encryption is active, but data from unencrypted agents will continue to be accepted.
Enforce: Encryption is active, and only encrypted data will be accepted.
It makes sense to begin with Enable initially. Once you are satisfied that all agents have been converted to encryption, switch over to Enforce, in order to identify hosts which still send data as clear text.
The encryption makes use of a shared password which must be stored on both the Checkmk-Server and on the agent as a simple text (‘Shared secret’), and which will be entered here. Choose a random password and have it ready for the second step — the configuration of the agent.
Create the file /etc/check_mk/encryption.cfg
with the following contents on the agent:
ENCRYPTED=yes
PASSPHRASE='XEwks9fm'
Of course use your own password in PASSPHRASE
. This file should definitely
be secured against reading by other users:
root@linux# chmod 600 /etc/check_mk/encryption.cfg
Now the following tests can be made (for information on these, see the article on Checkmk’s command line):
An invocation of
check_mk_agent
on the target system should produce a nonsense jumble of characters.A
telnet myhost123 6556
from the Checkmk-server should produce the same jumble.A
cmk -d myshost123
on the Checkmk-server should show the normal clear text.
Implementing with the Bakery
The implementation of the encryption with the agent bakery is very simple.
By creating the rules as just described, one is basically finished. Now only new
agents need to be baked and distributed. The file
/etc/check_mk/encryption.cfg
will be automatically generated and added to the agent package.
7. Monitoring Linux over SNMP
Since an easily-installed SNMP-Agenten is available for Linux, the question arises whether it is possible or even desirable to monitor Linux over SNMP? The answer is quite simple: possible yes, sensible no. Why is this so?
The monitoring data from the SNMP-agents is very limited, so that in any case Checkmk-agents are required for a halfway adequate monitoring.
The SNMP-agent delivers no useful data that isn’t already provided by the Checkmk-Agent.
The SNMP-agent is more time-consuming to install.
Last but not least, the SNMP protocol consumes far more CPU and network resources than a normal Checkmk monitoring.
There are however a few situations in which monitoring per SNMP in addition to the normal agents can be sensible. Such a case would be when either a user’s software, or a hardware monitoring tool supplied by the server’s manufacturer only delivers data via SNMP.
For such a situation, at the host’s characteristics in the Data sources submenu in WATO, set the Check_MK Agent to one of the options starting with Normal Checkmk agent and the option SNMP to either SNMP v2 or v3 or SNMP v1. Services that are accessible over SNMP as well as per Checkmk-agent (e.g. CPU-load, file systems, network cards) will be automatically retrieved by Checkmk-agent and not over SNMP. In this way a duplicated data transfer will be automatically prevented.
8. Hardware monitoring
8.1. Fundamentals
To be as comprehensive as possible the monitoring of a Linux-server must of course include the hardware. This is achieved in part directly with the Checkmk-agents, and partly also using special plug-ins. Additionally, there are also cases in which monitoring can be implemented per SNMP, or even over a separate management board.
8.2. Monitoring the SMART-values
Modern hard drives almost always use S.M.A.R.T. (Self-Monitoring,
Analysis and Reporting Technology). This system continuously logs data on the condition
of the HDD or SSD, and with the smart
plug-in Checkmk can retrieve
these values and evaluate the most important of them. In order that the plug-in
functions after an installation, the following preconditions must be met:
The
smartmontools
packet must be installed. This can be installed on all modern distributions with the appropriate packet manager.If the drives are connected via a RAID-controller and this allows access to the SMART-values, the appropriate tool must also be installed.
tw_cli
(3ware) andmegacli
(LSI) are supported.
If these prerequisites have been satisfied, and the plug-in has been installed, the data will be automatically read and appended to the agent’s output. In Checkmk the new services can then also be activated directly:

8.3. Monitoring with the help of IPMI
IPMI (Intelligent Platform Management Interface) is an interface for hardware management which, among other functions, enables the monitoring of hardware. Checkmk uses freeipmi for this, in order to access the hardware directly and without a network. It is installed from the package source and is ready for immediate use, so that the data will be transmitted at the next polling by Checkmk.
If freeipmi
is not available, or there are other grounds preventing an
installation, ipmitool
can also be used. This is often already present
on a system and must only served by an IPMI hardware driver — which can be provided
by the openipmi
package, for example.
Likewise, nothing more needs to be done here subsequently. The data will be
recorded by Checkmk automatically.
For error diagnosis the tools can also be executed manually in a host shell.
If the freeipmi
packet has been installed it can be used to control the function:
root@linux# ipmi-sensors Temperature
32 Temperature_Ambient 20.00_C_(1.00/42.00) [OK]
96 Temperature_Systemboard 23.00_C_(1.00/65.00) [OK]
160 Temperature_CPU_1 31.00_C_(1.00/90.00) [OK]
224 Temperature_CPU_2 NA(1.00/78.00) [Unknown]
288 Temperature_DIMM-1A 54.00_C_(NA/115.00) [OK]
352 Temperature_DIMM-1B 56.00_C_(NA/115.00) [OK]
416 Temperature_DIMM-2A NA(NA/115.00) [Unknown]
480 Temperature_DIMM-2B NA(NA/115.00) [Unknown]
Once ipmitool
has been installed the data output can be checked
with the following command:
root@linux# ipmitool sensor list
UID_Light 0.000 unspecified ok na na 0.000 na na na
Int._Health_LED 0.000 unspecified ok na na 0.000 na na na
Ext._Health_LED 0.000 unspecified ok na na 0.000 na na na
Power_Supply_1 0.000 unspecified nc na na 0.000 na na na
Fan_Block_1 34.888 unspecified nc na na 75.264 na na na
Fan_Block_2 29.792 unspecified nc na na 75.264 na na na
Temp_1 39.000 degrees_C ok na na -64.000 na na na
Temp_2 16.000 degrees_C ok na na -64.000 na na na
Power_Meter 180.000 Watts cr na na 384.00
8.4. Manufacturer-specific Tools
Many server producers offer their own tools for recording hardware information and delivering this via SNMP. The following prerequisites apply to be able to retrieve this data and provide it to Checkmk:
An SNMP-server has been installed on the Linux host.
The manufacturer’s tool has been installed — e.g., Dell’s OpenManage or Supermicros SuperDoctor.
The host has been configured in Checkmk for an additional monitoring per SNMP.
The new services for hardware monitoring thereby supported will then be automatically recognised. No further plug-ins will be required.
8.5. Supplementary monitoring over the Management Board
A management board can be configured for every host and additional data collected per SNMP. The services thereby recognised will likewise be assigned to the host.
Setting up a management board is very easy. In the host’s attributes, just enter the protocol, the IP-address and the access data for SNMP, and save the new settings:

With a service discovery the newly-discovered services will then be activated as usual.
9. Files and directories
9.1. Data paths on the monitored host
Path | Function |
---|---|
/usr/bin/check_mk_agent | Installation location of the Checkmk agent on the target system. |
/usr/lib/check_mk_agent | The home directory for agent extensions |
/usr/lib/check_mk_agent/plugins | Plug-ins which extend the agent for collecting supplementary monitoring data. Plug-ins can be written in any available programming language. |
/usr/lib/check_mk_agent/local | The user’s own ‘local checks’. |
/var/lib/check_mk_agent | Data location of the Checkmk agent on the target system. |
/var/lib/check_mk_agent/cache | Location of cached data from particular sections. These will be attached to the agent as long as the cache data is valid. |
/var/lib/check_mk_agent/job | Location of job data. These files will be added to the output on every agent execution. |
/var/lib/check_mk_agent/spool | Contains data created by e.g., cron jobs, and that includes its own section. These are also appended to the agent output. |
/etc/check_mk | Location of configuration files for the agents. |
/etc/check_mk/mrpe.cfg | Configuration file for MRPE — for the execution of standard, Nagios-compatible check plug-ins. |
/etc/check_mk/encryption.cfg | Configuration for the encryption of the agent data. |
/etc/xinetd.d/check-mk-agent | Configuration for the |