In recent years the container concept has taken the IT world by storm. That has of course thrown up questions about the monitoring of such containers. Checkmk can monitor Docker containers directly via the Linux agent. But Checkmk monitors not only the general status of the daemon or the container, but also the container itself. A full list of the elements that can currently be monitored can be found in the Catalogue of the Check plug-ins.
Alongside the status and inventory information which Checkmk can determine over the node (docker-jargon for ‘the host on which the containers are running’), Checkmk can also determine detailed status information for the containers. For this every container has to be added as a separate host in Checkmk if the container is to be monitored. Its data will be piggybacked from the node to this host.
From Version 1.6.0 of the Checkmk Enterprise Editions, container hosts can be automatically created or removed using the dynamic configuration.
2.1. Installation of agent and plug-ins
To be able to monitor a Docker node with Checkmk, it must first be monitored with the normal Linux Agents. This will as usual give you a basic monitoring of the host system, however there will be no information about the Docker daemon or about the container.
You will need the the
Install the plug-in as usual to
In the Checkmk Enterprise Editions you can of course also do this with the Agent Bakery,
which comes with the appropriate Docker node and containers rule set.
Please note, since version 1.6.0 the
docker Python library is required
not docker-py). At least version 2.0.0 is necessary (you can easily
check this by entering
python on the command line):
root@linux# python Python 2.7.16 (default, Sep 24 2020, 22:49:21) [GCC 8.2.0] on linux2 Type ‘help’, ‘copyright’, ‘credits’ or ‘license’ for more information. >>> import docker >>> docker.version '4.0.2'
If required you can install the library with
pip as root user:
root@linux# pip install docker
For nodes without Internet connection, copy the library and install it offline:
root@linux# pip install docker-4-3-1.tar.gz
Some distributions also allow installation via their package managers,
apt-get install python-docker under Debian, Ubuntu
and their derivatives. However, it can happen that too old, incompatible
versions are installed (for example version 1.9 under Debian 9).
In such cases, proceed as described above with
Attention: The packages,
must not be installed.
These make an outdated and incompatible version of the Docker library available under the same namespace!
docker-py (or both variants) have been installed,
a single uninstall is not enough because
pip cannot fix the namespace.
In this case, to ensure that the correct version is installed, please execute the following commands:
root@linux# pip uninstall docker-py docker root@linux# pip install docker
At this point you can now test if the plugin runs and outputs information and if the output is also available in the agent. You can test the plugin itself on the host:
root@linux# python /usr/lib/check_mk_agent/plugins/mk_docker.py
And you can test the output of the agent on the Checkmk server:
strings tmp/check_mk/cache/mydockerhost | grep "<<<docker"
If you now perform service discovery in WATO and activate the changes, you should find some new services that affect the Docker node itself (here from Version 1.6.0):
2.2. Finetuning the plug-in
As of Version 1.6.0 you can configure different parameters of the plug-in.
For example you can save resources by deactivating unnecessary sections or, if required,
by customizing the Docker API Engine endpoint (the default is the Unix socket
As usual, create the configuration file
/etc/check_mk/docker.cfg. A template with detailed
explanations can be found in the Checkmk directory
In the Enterprise Editions you can easily configure all parameters with the Agent Bakery.
2.3. Monitoring the container
Creating the container hosts
Of course the interesting aspect is the monitoring of the Docker containers. This will be implemented automatically by installing the plug-ins, however the services will not be assigned to the docker node, rather Checkmk assumes a single host per docker container.
The mechanism used here is called piggyback:
The plug-in or special agent transports data of other hosts — ‘piggybacked’ so to speak — alongside its own data. Checkmk stores this data in the directory
in a separate subdirectory. You only have to create hosts in WATO which have
the same name as the corresponding Piggyback subdirectories. Services are
then automatically assigned to these.
Since version 1.6.0 of the Enterprise Editions you can have these hosts created automatically. Use the Piggyback connector in the dynamic configuration.
Please note the following, if you create the hosts manually:
The host name must exactly match the directory created in
tmp/check_mk/piggyback. By default, this is the 12-character short ID of the container (for example,
2ed23056480f). However, you can also change the full ID or name as described in following chapter.
If the containers do not have their own IP addresses (which is usually the case), set IP-Address-Family to No IP.
For Data sources be sure to set Check_MK Agent to No agent.
You can set the Parent field to the host name of the Docker node.
It is also important that the Docker node and its container are monitored from the same Checkmk instance.
Once the container hosts have been created, and after performing a service discovery, new services appear on these. There you will also find a reference to the Piggyback origin:
Alternative names for container hosts
By default — as mentioned above — the 12-character short ID for the
container is used as the name for the container host.
This can optionally be configured differently. To do this, in the configuration file
container_id option to
in order to use the complete container ID as the name, or to
name in order to use the container name.
Checkmk Enterprise Editions users can of course do this in the Agent Bakery
Incidentally: With the Access to agents > General settings > Hostname translation for piggybacked hosts rule set you can define quite flexible rules with which hostnames — which are contained in the piggyback data — are converted to generate better host names for Checkmk. With this method you can also solve the problem of having containers with the same name on two different Docker nodes, for example. Using appropriate translation rules you could then, e.g., add a prefix to the names to make them explicit.
Monitoring the host’s status
Since a container’s host status cannot really be verified using TCP-Packets or ICMP, this must be determined in another way. The Docker container status service facilitates this — in any case it checks whether or not the container is running, and can thus be used as a secure tool for detecting the host’s status. Define a rule in the Host Check Command rule set for this purpose, and set the Use the status of the service option to the mentioned service. Don’t forget to set the conditions so that only containers are affected. In our example all containers are located in a folder with the same name:
Monitoring services in the container
To monitor details in the container itself (e.g., running processes, databases, log files, etc.), it is necessary that the Checkmk agent is executed in the container itself.
Exception: the three plugins
(disk I/O) also work without agents in the container and
are then calculated by the Checkmk agent on the node itself. Accordingly, agents running in containers
do not (again) output these values, because they know that they are running in containers.
However, even with an agent in the container, the call is still bundled via the docker node using the piggyback method. The container agent is simply called up by the node agent.
The agent installed in the container will of course only work if all necessary commands are available in that container.
Especially in the case of minimally built containers based on Alpine Linux, it may well be that elementary things like the bash are not present. In this case, you should monitor the container from the docker node — even if not all details can be recorded this way. The above mentioned use of the ruleset Host Check Command for monitoring the host status is only required if the container is not pingable.
3. Diagnostic options
3.1. Diagnosis of a Docker node
Should the setup not be successful, there are a number of options for analysing the problem. Verify therefore that an agent with at least this or a later version is installed on the host, for example with:
cmk -n myhost | grep -o -P '\[agent\] Version:.*?,'
Then you should make sure that the plugin — completely independent of the agent — works:
root@linux# python /usr/lib/check_mk_agent/plugins/mk_docker.py
If the version of the agent on the host is suitable and the plugin works, next check if the data is present in the agent’s output. The output can be downloaded as text data using the Download agent output option of the host’s dropdown menu in the GUI:
Alternatively, you could search the agent cache directly. For clarity the output in the following example is abreviated to the output for the node:
strings tmp/check_mk/cache/mydockerhost | grep "<<<docker" <<<docker_node_info>>> <<<docker_node_disk_usage:sep(44)>>> <<<docker_node_images>>> <<<docker_node_network:sep(0)>>>
If the sections are not shown here, the Docker installation will not be recognised.
The service Docker node info uses the following command, with which you can also manually test if your Docker installation is running correctly:
root@linux# docker info 2>&1
A typical error source, especially if Docker installation, agent and its plugin are running correctly one by one, is the python library for Docker mentioned above. If the described reinstallation does not solve the problem, check for further error messages when importing into Python:
root@linux# python >>> import docker
Example: Under Debian 9 there may be problems with the SSL backports.
These can be solved by uninstalling via
pip and reinstalling via
root@linux# pip uninstall backports.ssl-match-hostname root@linux# apt-get install backports.ssl-match-hostname
3.2. Diagnosis for a container host
If the container host receives no data, or respectively, no services are detected, first check if piggyback data is available for this host. The host’s name must be identical to the ID of the container. Alternatively, you can also make a manual assignment using the hostname translation for piggybacked hosts rule set. Here, however, only the Explicit hostname mapping option is available:
To verify whether piggyback data will be created for an ID, you can use the following command:
ls -l tmp/check_mk/piggyback/ 76adfc5a7794 f0bced2c8c96 bf9b3b853834
4. Host labels
Since version 1.6.0 of Checkmk there are so-called Host Labels.
The redesigned Docker monitoring automatically sets the three labels
cmk/docker_image_name and ` cmk/docker_image_version`.
You can use these labels, e.g. in conditions for your rules,
to make your monitoring configuration dependent on the image used in a container.
5. Special aspects in version 1.5.0
Docker monitoring was introduced in Checkmk version 1.5.0 and massively redesigned in version 1.6.0. If you are still using version 1.5.0, there are a few points to consider which differ from the current status documented up to here.
Plugins: mk_docker_node and mk_docker_container_piggybacked (instead of mk_docker.py)
Rule sets for agent baking: Docker node and Piggybacked Docker containers (instead of Docker node and containers)
If no agent is installed in the container, the agent of the docker node is executed in the container.
6. Files and directories
WATO stores the piggyback data here. For each host a subfolder with the host’s name will be generated. This contains a text file with the host’s data. The filename is the host that supplied the data.
Here the most recent agent output from all hosts is saved temporarily. The contents of a host’s file is identical to the output of the