 |
This article still shows the status of version 2.3.0. It is currently being prepared for publication - with the changes of version 2.4.0. |
1. Introduction
It is becoming increasingly common in cloud and container environments that hosts to be monitored can not only be generated but also expire automatically. Keeping up to date with the monitoring’s configuration in such an environment is no longer possible manually. Classic infrastructures such as for example, VMware clusters can also be very dynamic, and even if manual care is still possible it is in any case cumbersome.
The commercial editions of Checkmk support you
in this process with a smart tool the Dynamic Configuration Daemon or DCD for short.
The dynamic host management means that, based on information from monitoring
AWS, Azure, Kubernetes, VMware
and other sources, hosts can be added to, and removed from the monitoring in a fully-automated procedure.
The DCD is very generic, and is not limited only to host creation. The DCD forms the basis for future extensions of Checkmk which will dynamically adjust the configuration. This can also mean the management of users, for example. For this purpose the DCD works with connections. Each connection can get information from a very specific type of source, and has its own specific configuration.
With special connections, it will become even easier in the future, to automatically take hosts into Checkmk from an existing CMDB.
2. Managing hosts with the DCD
2.1. The piggyback connection
Currently, the DCD from Checkmk is only equipped with a single connection: the one used for piggyback data. This is very universal, since the piggyback mechanism is used by Checkmk in all situations where the query from a host (usually by special agent) provides data of other hosts (usually virtual machines or cloud objects).
Here are a couple of examples in which Checkmk uses piggyback in the monitoring:
In all of these cases the monitoring automatically retrieves data from other hosts (for example, the VMs) which are not contacted directly via the network and on which also no Checkmk agent needs to run. With the DCD you can add and also remove such hosts automatically in the monitoring so as to always reflect the real situation in a timely manner.
To do this the DCD analyzes the existing piggyback data and compares it to the hosts which already exist in the Setup, and then re-creates any missing hosts, or respectively, removes redundant ones. There are hosts which are automatically created by the DCD but which are still editable for you in the Setup-GUI.
2.2. The automation user
By default, the automation user exists in Checkmk. This user is created by Checkmk to enable automated fetches. An existing automation user is also mandatory for dynamic host management.
If, for whatever reason, you have deleted or changed this user in your system, you have to establish another user with automated access to the REST API. In this case, open the Connection to the REST API global setting in Setup > General > Global settings > Dynamic configuration.
Here you can enter a different user and their access data (name, password) instead of the automation user selected by default.
As soon as you have saved the changes with 
Save, the selected user is used to access the REST-API and you can continue with the dynamic host management.
2.3. Setting-up dynamic host management
Is piggyback data present?
The only requirement to be able to use the DCD is to have piggyback data. You will always have this data if you have correctly set up the monitoring of AWS, Azure and Co.
You can easily verify that via the command line as well, because the piggyback data from Checkmk will have been created in the ~/tmp/check_mk/piggyback directory:
OMD[mysite]:~$ ls tmp/check_mk/piggyback
myvm01 myvm02 myvm03If this directory is not empty, piggyback data has been generated in this site.
General connection settings
The Setup > Hosts > Dynamic host management menu entry brings you to the host management respectively its connections:
Create a new connection with 
Add connection.
The first part of the configuration is the General properties:
Here you assign, as so often, a unique ID and a title for this connection. Also important is the selection of the Checkmk site on which this connection should run. Because piggyback data is always processed locally, the connection must always be assigned to a specific site.
Properties of the connection
The second part is the Connection properties:
The connection type Piggyback data is already preselected here (and is currently the only one possible).
The Sync interval determines how often the connection should search for new hosts. If you keep the regular check interval of one minute, it makes no sense to do that much more often, since a piggyback data change can take place once a minute at most. In very dynamic environments you can use both check interval as well as the connection interval set to much lower values. However this also results in a higher CPU utilization on the Checkmk server.
Now it is important to add under Piggyback creation options at least one element (Add new element). This will take you to the settings for automatically created hosts:
Here you can specify two important things: In which folder the hosts should be created (here for example AWS Cloud 02), and which host attributes should be set (for the latter you must have Show more mode enabled)). Four important attributes are preset which are mostly applicable for piggybacked hosts:
No monitoring via SNMP.
No Checkmk agent on the host itself (data comes via piggyback).
Piggyback data is always expected (and there is an error if it is missing).
The hosts do not have an IP address.
Important: Only if you enable Delete vanished hosts will hosts be deleted when they disappear from your dynamic environment.
If you do not want to automatically create all hosts, you can do this by restricting the Only add matching hosts option with a regular expression. Important: here we mean the hosts that are being created, and not the hosts you have set up to monitor AWS, for example.
The latter can be achieved with the Restrict source hosts option. This refers to the names of the hosts that generate piggyback data.
Activate changes
Two further options deal with the automatic activation of changes – for the case that hosts really have been created or removed, since only then will they appear in the monitoring.
If activate changes takes a long time on your site, you can use Group "Activate changes" to make sure that it does not start immediately with each new host, but rather once a few hosts have been ‘collected’.
Furthermore, you can also completely stop the automatic activation of changes for specified times during the day — for example, for the times when your monitoring system is being actively looked-after. Because if the DCD activates changes, all other changes that you or a colleague have just made will also become active!
After saving the connection appears in the list.
It can however not yet run before you have activated the changes — only then does it start functioning.
So therefore do not be irritated by the message which appears right after saving:
Failed to get the status from DCD (The connection 'piggy01' does not exist)
3. Starting the connection
3.1. The first activation
After saving the connection properties, and following an activate changes, the connection will automatically start its operation. This can go so quickly that right after activating the changes you will immediately see how hosts are being created in monitoring:
If you reload this page shortly afterwards, these changes will probably have already disappeared, because they were automatically activated by the DCD. The new hosts are already in the monitoring and will be regularly monitored.
4. Automatic deletion of hosts
4.1. When are hosts being deleted?
As mentioned above, you can of course allow hosts which ‘no longer exist’ to be deleted automatically from monitoring by the DCD. That sounds at first very logical. What exactly is meant by ‘no longer exists’ is however at second glance a bit more complex, as there are several situations to be considered. In the following overview we assume that you have enabled the delete option — since otherwise hosts will never be removed automatically.
| Situation | What happens? |
|---|---|
A DCD connection is removed. |
If you shut down a DCD connection (do not activate this dynamic configuration connection), or remove it altogether, all hosts created by this connection are preserved. If necessary you must delete them by hand. |
A piggybacked host will no longer be monitored. |
If you remove the host from which you monitor your cloud or container environment from monitoring, it will of course generate no further piggyback data. In this case the automatically-generated hosts will after one hour be automatically deleted. |
A piggybacked host cannot be contacted. |
If your cloud environment is unreachable and the Check_MK service requesting it goes to CRIT, the generated hosts will remain in monitoring indefinitely. There is no one-hour timeout here! |
The Checkmk server itself is stopped. |
Stopping all monitoring will cause piggyback data to become obsolete, but of course this will not result in created hosts being deleted. The same applies if the Checkmk server is rebooted (which causes a temporarily loss of all piggyback data since these are in the RAM). |
A host is no longer in the piggyback data. |
This is a normal situation: A host in a cloud/container environment has disappeared. In this case it will be immediately removed from monitoring. |
Note that with the Automatic host removal rule, there is an option for all hosts to have them automatically removed. Both options for lifecycle management work independently of each other, i.e. a host is removed if one of the two conditions is met.
4.2. Configuration options
In addition to the question of whether hosts should be removed automatically at all, in the connection properties there are three more options that affect the deletion — options which we skipped discussing earlier:
The first setting – Prevent host deletion right after initialization – affects a complete reboot of the Checkmk server itself. In this situation piggyback data for all hosts will at first be missing until the hosts are queried for the first time. To avoid the senseless deletion and reappearance of hosts (which is also accompanied by repeated notifications for known problems), deletions will by default be generally waived during the first 10 minutes. This time limit can be customized here.
The Validity of missing data option handles the situation where a host, whose monitoring data created several hosts automatically, returns no piggyback data. This can be the case, e.g. when access to AWS and Co. has stopped working. Or also of course if you have removed the special agent from the configuration. The automatically-generated hosts will remain for the set time in the system before being removed from the Setup.
The Validity of outdated data option is similar, but treats the case that even if piggyback data is being received, but not from some hosts. This is the normal case if, e.g. virtual machines or cloud services are no longer available. If you want the corresponding objects to disappear from Checkmk in a timely manner, then set a correspondingly short time span here.
5. Diagnoses
5.1. Execution history
If you want to watch the DCD at work, for each entry in the list of connections you will find the 
icon.
This takes you to the execution history:
If for some reason the creation of a host fails, you will see this in the execution history.
5.2. Audit log
With Setup > General > Audit log you open a page with the list of all changes that have been made in the Setup GUI — regardless of whether they have already been activated or not.
Look for entries from the automation user or the user that you have authorized instead (see the The automation user section).
The DCD works under this user account and generates changes there — so here you can follow which hosts the DCD has created or removed, and when.
5.3. DCD log file
The DCD’s log file is ~/var/log/dcd.log.
Here is an example which fits the previous image:
2021-11-10 14:45:22,916 [20] [cmk.dcd] ---------------------------------------------------
2021-11-10 14:45:22,916 [20] [cmk.dcd] Dynamic Configuration Daemon (2.0.0p14) starting (Site: mysite, PID: 7450)...
2021-11-10 14:45:22,917 [20] [cmk.dcd.ConnectionManager] Initializing 0 connections
2021-11-10 14:45:22,918 [20] [cmk.dcd.ConnectionManager] Initialized all connections
2021-11-10 14:45:22,943 [20] [cmk.dcd.CommandManager] Starting up
2021-11-10 15:10:58,271 [20] [cmk.dcd.Manager] Reloading configuration
2021-11-10 15:10:58,272 [20] [cmk.dcd.ConnectionManager] Initializing 1 connections
2021-11-10 15:10:58,272 [20] [cmk.dcd.ConnectionManager] Initializing connection 'piggy01'
2021-11-10 15:10:58,272 [20] [cmk.dcd.ConnectionManager] Initialized all connections
2021-11-10 15:10:58,272 [20] [cmk.dcd.ConnectionManager] Starting new connections
2021-11-10 15:10:58,272 [20] [cmk.dcd.piggy01] Starting up
2021-11-10 15:10:58,273 [20] [cmk.dcd.ConnectionManager] Started all connections6. Files and directories
| Path | Function |
|---|---|
|
Piggyback data is created here. A directory is created for each piggybacked host included in the piggyback data. |
|
The log file for the Dynamic Configuration Daemon (DCD). |