Automatic agent updates - Distribute agents and plug-ins automatically

The CEE Checkmk Enterprise Editions can update their agents on Linux, Windows and Solaris automatically. With it you can not only easily update the agents in the case of new Checkmk versions, even a changed configuration of the agents can be applied in this way. In this way you can take advantage of the Agent Bakery function to apply individual configurations to hosts.

1. Setting up automatic updates

The automatic deployment of agents is by default globally disabled. So before you take care of the configuration itself, enable the Activation of automatic agent updates option under Setup > General > Global Settings > Automatic Agent Updates.

agent deployment activation of automatic agent updates

To implement the updates, follow these steps: First open Setup > Agents > Windows, Linux, Solaris, AIX and select Agents > Automatic updates.

agent deployment automatic agent updates

See Prerequisites for a list of prerequisites that need to be met for the automatic updates to work correctly. You can simply tick these off in order. Please do not forget that you can get more information for each of these items via Help > Show inline help. Clicking on icon edit will take you directly to the setting you need to configure. Specifically, the settings described in the following five chapters must be made and activated via the Master switch.

1.1. Creating a signature key

The system is designed in a way that the agents can download updates via HTTP or HTTPS from their central monitoring server. Because agents contain executable code it is especially important to guard against the possibility of falsified agents coming from an attacker. Signature keys are used for this purpose. These keys consist of a pair of public and private keys (the public-key method).

You can create as many signature keys as you like. Each of these is secured with a passphrase, which you will subsequently need to enter each time you sign. This prevents, for example, an attacker gaining access to a backup of the Monitoring server which agents could sign.

Create a signature key here and record the passphrase. This passphrase can later neither be changed nor restored. If the key is lost, it can mean that all agents need to be updated manually once again!

1.2. Configuring the Update plug-in

The actual update is performed by an agent plug-in named cmk-update-agent. This is done by the agent in a definable cycle (e.g. once per hour). At this time the agent asks the deployment server (your central monitoring system) if there is a new package available for this host, and if so it will then perform an update.

The plug-in must of course be present and correctly configured in the agent. To do this, extend the agents using the Agent updater (Linux, Windows) rule set around this plug-in. Please note that the rule set here follows the ‘First rule per parameter wins’ principle. This allows you to define basic settings in general so that they do not have to be reset again and again in the more specific rules. In addition, of course, the online help (Help > Show inline help) provides more information about each item as soon as you activate it.

Below are a few explanations of the individual points:

Activation

This setting must be enabled (Deploy plugin …) to allow the plug-in to be added to the agent. Here, for example, rule inheritance can be used to set a default in a higher-level folder and override this for individual hosts/folders as needed.

Update server information

Optionally, data for the connection of the agent updater to the Checkmk server can already be entered here for configuration. If this entry is not configured, the information must be entered later when registering the agent updater.

Usage

In the case of distributed monitoring with a centralized configuration, the agent updater receives its responsible update server by default from the Checkmk site to which it connects. This option can be used to configure that the update server entered here is used permanently and the automatic update server is ignored.

DNS name or IP address of the update server

Here you enter the DNS name under which the Checkmk server is accessible. It is important here that the host to be monitored can resolve this name and that it is configured as a host in Checkmk. Take care when using HTTPS that the name of the certificate matches the name of the Checkmk server as known to the host.

Name of the Checkmk site of the update server

Here you enter — with a few exceptions — the name of the site on which you are currently configuring this rule. An exception to this approach would be if the affected hosts should be ‘moved’ to another Checkmk site. In such a case, for one time only, enter a different site here. See also below under Application scenarios. In a distributed setup with a central configuration, the site where you want to register may also be different from the central site where this rule is configured.

Protocol for fetching updates

If — as we recommend — you use HTTPS you must ensure that the agent updater also has a CA certificate available to verify the connection.

Important: Depending on the configuration of the server, HTTPS (including forwarding from HTTP to HTTPS) may be forced. In such a case, configuring this rule to HTTP has no effect.

Encrypted communication for updates

CA or self-signed certificates configured here are available to the agent updater for the verification of HTTPS connections. Alternatively, in the case of a distributed setup with a central configuration, certificates can also be made available to the agent updater from the Checkmk site, or these can be imported directly during the connection via command line parameters.

Important: If the server’s certificate chain is signed with a public CA, the connection can normally be verified without imported certificates. However, as soon as imported certificates from one of the mentioned sources are available to the agent updater, all other CA certificate authorities are ignored! In the case of problems with the configuration of HTTPS, please consult the following FAQ below.

Interval for update check

Here you specify the interval in which the agent updater queries the configured monitoring server whether any updates are available. As long as the specified interval has not expired, a cached call is returned, so as to burden the network load as little as possible. It usually makes sense to use an interval of not less than 10 minutes, otherwise there is the very great danger that your network will be very heavily burdoned if you have a large number of Checkmk agents. If you do not set a value here a default value of 60 minutes will apply.

Proxy settings

This rule setting is likewise optional. The agent updater assumes that there is a direct connection to the Checkmk server and it will ignore all local proxy settings — even if proxy settings have actually been configured on the destination host. If this is the desired behavior this rule setting can therefore be omitted. Otherwise either enter proxy settings manually, or use the host’s existing environment variables.

Executable format (Linux)

You can optionally specify how the plug-in is added to the installation package for the agent. How the rule behaves by default depends on the target system:

Linux (deb, rpm, tgz): You do not have to manually adjust anything for these systems; the agent updater is passed as a 64bit binary. You can also optionally select a 32bit binary for legacy systems, or the old Python script. Important: For the binary you will need the glibc package (minimum the 2.5 version). Linux distributions have generally met these requirements since 2006.
Windows: For Windows hosts Checkmk will always deploy a 32bit-executable. This rule is being ignored for Windows hosts. Note: Should you find a 64bit binary of the agent updater on any of your Windows hosts, this version dates back to an older version of Checkmk and is not up-to-date.
Solaris: You do not have to adjust anything here either. Checkmk will use the Python script even if you leave the default value on the 64bit binary.
Other architectures: If you have packages for other architectures such as arm or ppc, set this option manually to Python, since Checkmk does not intercept this automatically and no binaries are offered for these platforms.

If you still need to rely on the old Python script the following requirements apply to the system:

Python3 in version 3.4 or newer
The Python-Modules requests, PySocks and pyOpenSSL

Signature keys

Select at least one signature key whose signature should be accepted by the agent updater. You can also optionally specify multiple keys. This can be the case if, for example, you want to disable an old key. For this purpose the host’s agent updater must in the interim accept both keys.

After this last setting, your rule could look like the following screenshot. Save all your entries by clicking on context button save 1px border .

1.3. Baking and signing agents

Next, you can immediately bake and sign the agents configured in this way in a single step. This is because the newly created and customized rules will not be found in the installation packages until you have created/baked them again. Navigate to Setup > Agents > Windows, Linux, Solaris AIX and click Setup > Agents > Windows, Linux, Solaris AIX, then click on Agents > Bake and sign agents. You must now enter the passphrase of the key you want to use. After you click Bake and sign, the baking procedure will start as a background process. When this process has completed, you will be informed:

agent deployment agent baking successful

All agents signed in this way will be assigned to a corresponding icon signature key symbol. If you have created multiple keys, sign with these additional keys separately.

Important: It is sufficient for the agent updater on the host to be monitored if the new package has been signed with one of the keys known to the updater.

1.4. Registering agents

In the next step register the hosts to be monitored on the Checkmk server. Since a new host is not yet trusted by the Checkmk server, and the server does not yet know that the host should be updated automatically, the agent must be installed manually — once-only — on the host. To do this download the icon agents package for the host from Setup > Agents > Windows, Linux, Solaris, AIX. Make sure that the package also contains the agent updater plug-in.

Now copy the package to the host and install it as usual with rpm, deb or msiexec (or with a double-click as applicable). Afterwards you will find the agent updater or the plug-in — depending on the host’s operating system — in the following directories:

Under Unix-type systems – in the path /usr/lib/check_mk_agent/plugins/[configured interval]/ (A script of the same name is also stored under /usr/bin, so that cmk-update-agent is also available as a command.)
Under Windows: The agent updater is integrated into the agent for Windows. You will find the agent at C:\Program Files (x86)\checkmk\service\. The updater itself can be controlled with the check_mk_agent.exe updater command.

Now call the agent updater with the register argument. Under Windows this must be done in a prompt with administrator rights. Enter the required information in sequence. If you have followed these instructions from the beginning so far and have installed a baked agent, not all of the following settings will be asked for. These are given here for reference purposes only.

With the following command, you now register the agent updater from a Linux host:

root@linux# cmk-update-agent register -v
-------------------------------------------------------------------
|                                                                   |
|  Check_MK Agent Updater v2.0.0 - Registration                     |
|                                                                   |
|  Activation of automatic agent updates. Your first step is to     |
|  register this host at your deployment server for agent updates.  |
|  For this step you need an administration account on WATO for     |
|  that server.                                                     |
|                                                                   |
-------------------------------------------------------------------
Deployment server to connect to:
mymonitoring.example.intern

Protocol to use for connection [http/https]:
https

Check_MK site on deployment server:
mysite

Our host name in the monitoring:
myhost

WATO user with admin permissions:
cmkadmin

Password:

Going to register agent at deployment server
Successfully registered agent of host "myserver" for deployment.
You can now update your agent by running 'cmk-update-agent -v'
Saved your registration settings to /etc/cmk-update-agent.state.

Hint: you can do this in scripts with the command:

cmk-update-agent register -s mymonitoring.example.intern -i heute -H myserver -p https -U cmkadmin -P * -v

As the agent updater for Windows hosts is integrated into the agent itself, the command for registering it looks a little bit different:

cd "\Program Files (x86)\checkmk\service"
check_mk_agent.exe updater register -s mymonitoring.example.intern -i mysite -H myhost -p https -U cmkadmin -P cmk -v

Alternatively, you can perform the registration in non-interactive mode by entering the required data via the command line option. A call to the cmk-update-agent register—help here shows the settable options. Noteworthy here is that the one-time registration can also be made via an Automation-User — in this method the user is as usual passed via -user/-U, and the automation secret is passed via -secret/-S.

Some notes about registration:

When registering, the plug-in also needs the name of the host as it is known in the monitoring. This is not necessarily identical to the host name of the computer. The host name is then stored locally together with the key.
To use HTTPS, HTTPS must be set up on your monitoring server. HTTP is much easier here, but does not provide encryption of the transmission. Since the agent can theoretically contain passwords, HTTPS is the recommended method. The authenticity of the agent is however ensured independently by the signature.
The login as Checkmk administrator is only required once. On registration the agent and the server agree a secret key known only to this host. The password of the Checkmk administrator is not stored anywhere.
While the interactive mode only polls fields that are not yet in any configuration, the non-interactive mode allows all fields displayed in the Help to be set and has the highest priority for this call. Options that are saved in cmk-update-agent.state will be overwritten, but options from cmk-update-agent.cfg will not be overwritten. See also below Viewing the Local Configuration.

Following a successful registration the key is stored at the agent in the /etc/cmk-update-agent.state file. On the server it is located in ~/var/check_mk/agent_deployment/myhost. From now on the key allows the host to download its own agents from the server without a password. It is not possible to download agents from other hosts, since these could contain confidential data.

1.5. Master switch

Finally, enable the agent by clicking icon edit at the Master switch. The table Prerequisites should now look like this:

agent deployment prerequisites fulfilled

From now on, the agent will report at the end of each configured update interval and look for a new version of the agent. If a new version is ready — and signed — it will be downloaded and installed automatically.

At the same time of course, the Master switch is also one way to deactivate the update process globally.

A step-by-step guide is also provided by the video which originated at the Checkmk Conference #3 (2017), under the following link. This is not the latest version — however the basic procedure has not changed: The new automatic agent updates

2. Restricting updates to specific hosts

Before rolling out a new agent to a large number of hosts, you will certainly want to first try it out with a smaller number of hosts. This important step prevents a possible mistake of serious dimensions.

For this function, use the middle box on the Automatic agent updates page:

After you have met the conditions for selecting hosts here, you can use the field Test with this host name to enter individual hostnames and check if the updates for these hosts have now been enabled or not. The conditions are always connected with and.

Important: On hosts that are not yet to be provided with automatic updates, the installed package must not contain the agent updater plug-in — otherwise the plug-in will regularly warn you that the host has not yet been registered.

3. Diagnoses

There are quite a few sources of information for diagnosing whether all updates work as intended:

3.1. Statistics on the automatic agent updates page

This overview shows how the individual hosts in the agent update behave. The online help (Help > Show inline help) gives further explanations. Clicking on icon view 20 provides a detailed list of the individual hosts. You can also get to the complete list of all registered hosts via the Monitor > Analyze > Agent update status view. There you can then search for specific individual hosts.

For an agent intended for a host (Target Agent) – which was last downloaded from the host (Downloaded Agent), and which is currently installed on the host (Installed Agent) – this list will also show documentation on how the agent’s hash begins. In this way you can always see if the specifications have been met or where the process is currently located. It should be noted here that the status information appears to the left directly in the communication between the Agent Bakery and agent updater, while the Update Check and Update Check Output fields come from the agent updater plug-in when querying the agents of the host, and that due to caching (defined by the polling interval) these may be updated at a different time.

3.2. The new Check_MK Agent on each relevant host

If you have installed the update plug-in on an agent, this will regularly produce the current status of the update in the form of monitoring data. The service discovery generates a new service from the host with the name Check_MK Agent. This again reflects the current state of the update. This way you can be notified of any problem with the updates.

This check’s state is limited to a severity of WARN.

3.3. Viewing the local configuration

The behavior of the agent updater is governed by the two files cmk-update-agent.cfg and cmk-update-agent.state. It always applies that defined values from the .cfg file override those from the .state file. If the agent updater shows unexpected behavior, it is sometimes worth having a look in the configuration. There is also a handy feature if you call the agent updater directly from the command line:

root@linux# cmk-update-agent show-config
Showing current configuration...

Configuration from config file (/etc/check_mk/cmk-update-agent.cfg):
signature_keys: ['-----BEGIN CERTIFICATE-----\ncertificate\n'-----END CERTIFICATE-----\n']
protocol: http
interval: 86400
site: mysite

server: 10.0.0.42
certificates: []

Configuration from state file (/etc/cmk-update-agent.state):
installed_aghash: a91310934c83ce696
last_error: 404 Client Error: Not Found for url: http://mymonitoring/myothersite/check_mk/deploy_agent.py
host_name: myhost
last_check: 1550232737.28
last_update: 1550232737.37
host_secret: lvhfstjgmblmutzrplkspwifmmfperlditvcqmrxglgzbeaeplibcthawgzsggou
user: automation

3.4. Log messages on the destination host itself

In the case of a problem you will also find log data for the updates on the host to be monitored. On Linux cmk-update-agent logs important information to syslog — such as warnings and errors. A more detailed log, including debug outputs and possible tracebacks can be found under /var/lib/check_mk_agent/cmk-update-agent.log. Likewise, under Windows a detailed log will also be in the file log/cmk-update-agent.log. Under both systems you can also use the command line option --logfile LOGFILE to specify an alternate path for a debugging log.

/var/log/syslog

Jul 02 13:59:23 klappgrill [cmk-update-agent] WARNING: Missing config file at ./cmk-update-agent.cfg. Configuration may be incomplete.
Jul 02 13:59:23 klappgrill [cmk-update-agent] ERROR: Not yet registered at deployment server. Please run 'cmk-update-agent register' first.

/var/lib/check_mk_agent/cmk-update-agent.log

2021-07-02 17:58:18,321 DEBUG: Starting Check_MK Agent Updater v2.0.0p9
2021-07-02 17:58:18,322 DEBUG: Successfully read /etc/cmk-update-agent.state.
2021-07-02 17:58:18,322 DEBUG: Successfully read /etc/check_mk/cmk-update-agent.cfg.
...
2021-07-02 17:58:18,387 INFO: Target state (from deployment server):
2021-07-02 17:58:18,387 INFO:   Agent Available:     True
2021-07-02 17:58:18,387 INFO:   Signatures:          1
2021-07-02 17:58:18,387 INFO:   Target Hash:         081b6bcc6102d94a
2021-07-02 17:58:18,387 INFO: Ignoring signature #1 for certificate: certificate is unknown.
2021-07-02 17:58:18,388 DEBUG: Caught Exception:
Traceback (most recent call last):
  File "/build/enterprise/agents/plugins/cmk_update_agent.py", line 1733, in main
  File "/build/enterprise/agents/plugins/cmk_update_agent.py", line 714, in run
  File "/build/enterprise/agents/plugins/cmk_update_agent.py", line 1372, in _run_mode
  File "/build/enterprise/agents/plugins/cmk_update_agent.py", line 1071, in _do_update_as_command
  File "/build/enterprise/agents/plugins/cmk_update_agent.py", line 1150, in _do_update_agent
  File "/build/enterprise/agents/plugins/cmk_update_agent.py", line 1221, in _check_signatures
Exception: No valid signature found.

4. Application scenarios

4.1. Deactivating automatic host updates

If a host is to be removed from the automatic updates, alter its setting with the Install agent updater (Linux, Windows) rule set so that the update plug-in is deactivated there. At the next regular update the agent itself then removes its own updater!

It goes without saying that the update can then only be reactivated by the manual installation of a new agent package! The registration is retained and does not have to be renewed.

4.2. Migrating to a new monitoring site

Should you want to move to a new Checkmk site in a single-site setup without losing the hosts registered on the server, it should be noted that for a successful agent update process the following information on server and host must match:

The name under which the host is monitored and registered
The Host Secret, which was automatically assigned during registration.
The signature used to sign the agents

To achieve this, follow these steps:

First add all hosts whose registration information is to be migrated to the new site to the monitoring. Make sure the hosts in the new site are monitored under the same name. Then copy the ~/var/check_mk/agent_deployment folder from the old to the new monitoring site.
Export the signature key(s) that are accepted by the agents installed on the hosts. The signature keys can be exported and imported using Setup > Agents > Windows, Linux, Solaris, AIX > Agents > Signature keys.
Import these signature keys from step 2 on the new monitoring site.
Configure the agent updater rule on the new monitoring site according to the instructions, and sign the baked agents with the imported signature key(s).
Lastly, in the agent updater rule on the old site, configure the fields for the update server and the name of the Checkmk site conforming to your new monitoring site, and bake the agents again. Note: Please check at this point that you have specified everything correctly before you re-bake the agents.

As soon as the next automatic updates go through the hosts, the old monitoring site will be locked out. From that time on the hosts to be monitored will only answer to the new Checkmk server. Following the second automatic update the agent will be installed by the new Checkmk server accordingly.

4.3. The agent updater as automatic installer

Attention: This is not an official feature of the agent updater. These instructions are therefore intended primarily for more experienced users. The official way to install a Checkmk Agent on a host is to download and run the agent package appropriate for the system. It is however also possible to allow the Checkmk Agent to be installed initially by the agent updater, since this also works as a stand-alone program.

Proceed as follows:

Copy the cmk-update-agent binary or the cmk_update_agent.py script to the host to be monitored (both can be found at ~/share/check_mk/agents/plugins on the Checkmk server).
Register the host on the Checkmk server by invoking cmk-update-agent register. Here it makes sense to pass the required registration information directly via the command line – especially if you want to use an installation script. The corresponding options can be displayed when calling cmk-update-agent register—help.
Then, with a final call to the agent updater plug-in, install the agent with all of the configuration details for the host being monitored. However since there is no local configuration (the agent updater also displays a corresponding warning), and thus no signature for the agent package to be downloaded, call the updater once more with cmk-update-agent—skip-signatures to explicitly trust the downloaded package. The prerequisite for the installation by agent updater is, of course, that the Agent Bakery has a suitable agent package ready for the target host on the Checkmk server.

5. Agent updates in distributed monitoring

Since Checkmk 2.0.0 it has also been possible to distribute baked agents via remote sites. The prerequisite for this is a setup with a central configuration and a connection which allows the central site to be reached from the remote sites via HTTP/HTTPS.

5.1. Functionality

Technically, the feature is implemented in such a way that update requests to a remote site are forwarded to the central site — so that the entire configuration as well as the baking of the agents takes place exclusively on the central site. Agent packages that have already been requested at a remote site are cached there (as long as they are valid), so that the packages do not have to be downloaded again from the central site. In addition, the requested data will be checked for consistency on the remote site, so that unnecessary connections to the central site are avoided.

Unlike in a single-site setup, the appropriate update server for the hosts does not originate exclusively from the set of rules of the agent updater, but is communicated to the requesting agent updater by the contacted Checkmk site. In this process, a host is provided with its server by the site from which it is being monitored. The specification of a Checkmk server is therefore only needed for the one-time registration, which can theoretically take place at any site — that is accessible from the host — in the entire distributed monitoring system. If the connection to the automatically determined server fails, the previously saved server (from the rule configuration or previously-entered manually during registration) is used as a fallback.

5.2. Configuration

The distribution of agent packages via remote sites does not have to be activated separately — the respective remote site recognises the situation automatically and communicates accordingly with the central site and the requesting agent updater. If the agents are to report explicitly only to the central site for updates, this can be done via a fixed update server in the rule set for the Agent Updater.

In order for the agent updates to actually work in distributed monitoring, however, some settings must be made on the central site, all of which can be found in Setup > General > Global Settings > Automatic Agent Updates. If the settings differ for each remote site, they can be edited in Setup > General > Distributed monitoring > icon site globals (Site specific global settings).

Connection to the central Agent Bakery

At this point, the URL through which the central site can be accessed from the remote site must be specified, including its protocol and the character string 'check_mk', for example https://myserver.org/mysite/check_mk. While the Checkmk site will try to identify all other missing settings itself, the specification of this URL is not optional, as this connection direction will otherwise not be established in the case of a central configuration. If the protocol is HTTPS, the remote site automatically uses the CA or self-signed certificates available in the setup for the verification of the connection Setup > General > Global Settings > Site management > Trusted certificate authorities for SSL. The remote site also requires a previously-created automation user to be able to log on to the central site. This can also be selected here. If none is specified, the remote site searches for an automation user with the name 'Automation'.

Connection to a remote Agent Bakery

Since the remote sites are not necessarily accessible from the respective monitored hosts via the same URL as from the central site, a URL can be configured here for this purpose. This URL is then automatically communicated to the host (or the requesting Agent Updater) when a connection is made to a Checkmk site. The configuration as a site-specific global setting makes particular sense here. If no URL is specified, it is assumed that the remote sites are accessible from the central site and from the monitored hosts under the identical URL. If it is an HTTPS connection, the appropriate certificate can also be automatically made available to the host. Since the central CA store cannot be used for this, appropriate certificates can be specified at this point. Alternatively, the certificates can also be specified in the Agent Updater rules.

6. HTTPS handling

At various points in this article, there are references to securing the respective connections via HTTPS. At this point, a central summary is given of everything that needs to be done for complete security via HTTPS. Both the connection from the remote site to the central site and from the host to the Checkmk site (regardless of whether it is a single-site setup or a distributed setup) can and should be secured via TLS, i.e. via HTTPS.

6.1. Using HTTPS

To be able to address a Checkmk site via HTTPS, the monitoring server must first be configured for HTTPS. This can be achieved, for example, through a suitable configuration of the system’s Apache, or in an especially simple way through the HTTPS settings on the Checkmk appliance.

Whether a Checkmk server is then addressed via HTTP or HTTPS is determined by the URL configured in each case. If this begins with 'https://', the server is also addressed accordingly via the HTTPS protocol and port 443. This also of course applies in the case of the explicitly-specified protocol when the server being contacted is the one from the Agent Updater configuration Update Server Information.

6.2. Providing certificates

In order for an HTTPS connection to be established, it must be possible to verify the certificate chain or self-signed certificate — depending on how the server is configured — for the server to be contacted, which is made possible by providing suitable CA or self-signed certificates. This can be realised as follows:

Host to Checkmk server connection

The agent — or more precisely, the Agent Updater — always attempts to verify HTTPS connections and terminates them if this is not possible. Certificates for verification are available to the Agent Updater from the following sources:

If the HTTPS setup of the server is a certificate chain that has been signed by a public CA (Certification Authority) such as Let’s Encrypt, this can probably be verified without further configuration, as the Agent Updater can access its own certification store for this purpose. This concept has its weaknesses however, since this store can only be as up-to-date as the software used — very new CAs or recently-revoked CAs can often not be taken into account. The internal CA store will therefore be ignored as soon as one or more certificates have been imported via one of the following mechanisms. Certificates from the following three sources are stored locally on the host and thus form their own certification store:

Using the rule entry Certificates for HTTPS verification, certificates can be included in the agent package and will be available to the Agent Updater following an installation or update.

When configuring the connection to the remote site via Setup > General > Global Setting > Automatic Agent Updates > Connection to remote agent bakery, certificates can be specified that are passed to the Agent Updater during the connection and with which the HTTPS connection to the respective remote site can be verified. This is particularly useful if it is not yet clear at configuration time which host is assigned to which site. This import option also reduces the number of agents that need to be baked in, as the correct certificates for the respective update servers do not have to be host-specifically configured.

The Agent Updater can be called with the '--trust-cert' command line argument. This attempts to retrieve a CA or self-signed certificate from the server and import it. Unfortunately, this only works when either a self-signed certificate is used (as this is already the server certificate) or when the CA certificate is stored on the server, i.e. is at the end of the certificate chain. This latter is generally not required for a valid certificate chain and is therefore unfortunately often not the case. Caution: If a certificate is imported in this way, the user themselves must ensure the authenticity of the server, as the certificate does not come from a separate source.

And if nothing else helps:

If no valid certificate is available to the Agent Updater at all, the HTTPS verification can be bypassed by using the command line argument '--insecure'' for a call. This can be useful if the valid certificate is only waiting to be retrieved by the server during the next connection, but the Agent Updater will be 'locked out' by this certificate being unavailable. Caution: HTTPS verification is actually fully-deactivated by this. Communication is nevertheless encrypted — the use of this argument is therefore 'better than nothing'.

Connection from a Remote Site to the Central Site

The distribution of certificates is simpler for the connection from the remote site to the central site, since in this situation the WATO world will not be 'exited', so to speak. The remote site can therefore use the WATO-internal Certification Store under Global Settings > Site Management > Trusted certificate authorities for SSL. It is thus sufficient to import the certificate(s) via the central site, or if necessary via the Site specific Global Settings if the central site is accessible through multiple URLs.

7. Troubleshooting

7.1. Typical errors and their solutions

Already fixed errors in the Checkmk Agent service

The agent updater will really only be run once within the update interval, so an error will be continuously-displayed until either you call the plug-in manually, or the next interval is pending.

Registration fails after a manual reinstallation of the Checkmk agent

The agent updater creates its own status file cmk-update-agent.state independently (under Linux/Unix in /etc, and under Windows in the config folder). This file remains on the host after a deinstallation, so that the registry information does not get lost. A new installation will find the file and continue using it. If this situation is undesirable, after a deinstallation simply delete the cmk-update-agent.state file manually.

Update status for hosts with no automatic updates active

The Monitor > Analyze > Agent Update Status page displays all of the hosts that are are in the monitoring and for which a status file exists on the Checkmk server. It does not matter if the host actually reports to the Checkmk server for automatic updates. Should an unexpected host be displayed here, it is worth taking a look in the /omd/sites/mysite/var/check_mk/agent_deployment folder — the cause will probably be an old or accidentally-created registry.

The connection over SSL/TLS does not function

The agent updater is designed to explicitly trust only the certificates which are usually specified under Agent updater (Linux, Windows) in the HTTPS configuration. In particular locally-installed certificates are ignored. It can also occur that the Checkmk server is accessible through the browser, while the agent updater cannot connect due to a wrong configuration.

In the HTTPS configuration of the agent updater rule a root certificate must be specified with which the connection to the Checkmk server can be verified. In other words: the certificate chain included in the Checkmk server’s server certificate must be verifiable by the certificate given here. Often the server certificate is specified here instead — this is however not suitable for this purpose.

Take a look at the Checkmk server’s certificate chain with the OpenSSL tool. Due to the chain’s length here only a relevant section is shown and the abbreviated sections marked by […]:

root@linux# openssl s_client -connect mymonitoring.example.net:443
[...]
subject=/CN=mymonitoring.example.net
issuer=/C=DE/O=Deutsche Telekom AG/OU=T-TeleSec Trust Center/CN=Deutsche Telekom Root CA 2
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3832 bytes and written 302 bytes
Verification: OK
---
[...]

For the last entry — in our case subject=/CN=mymonitoring.example.net — you need a valid root certificate. This must not necessarily be — as in this example — the issuer of the certificate. It will usually be a chain of issuers.

Then look at the certificate used. Here too due to the length it will be shortened as seen above:

root@linux# openssl x509 -text -noout -in myca.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 38 (0x26)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C = DE, O = Deutsche Telekom AG, OU = T-TeleSec Trust Center, CN = Deutsche Telekom Root CA 2
        Validity
            Not Before: Jul  9 12:11:00 1999 GMT
            Not After : Jul  9 23:59:00 2019 GMT
        Subject: C = DE, O = Deutsche Telekom AG, OU = T-TeleSec Trust Center, CN = Deutsche Telekom Root CA 2
        [...]
        X509v3 extensions:
            [...]
            X509v3 Basic Constraints:
                CA:TRUE, pathlen:5
            [...]

The top certificate — seen in the above excerpt — is not permitted to have a dependency on another certificate. You can recognize that the issuer (Issuer) and the item (Subject) are identical and that the option CA:TRUE is included. In addition the issuer’s chain that authenticates an object must be consistent until the last entry. You therefore also need all intermediate certificates if the issuer of the last certificate should not be a CA.

A detailed insight into this whole topic is also provided by the following video, which was created at the Checkmk conference #4 (2018): SSL and Certificates

Error message: Cannot open self cmk-update-agent or archive cmk-update-agent.pkg

On some Linux systems the program Prelink is installed and a cronjob is activated which regularly examines all binary files on the system, and adapts them if necessary to speed up the programs. However the Agent Updater plug-in is packaged with the PyInstaller program which is not compatible with such actions, and is therefore broken. Checkmk therefore has a blacklist entry for deb/rpm packages which is stored under /etc/prelink.conf.d, and — if prelink exists — sets an entry in the existing /etc/prelink.conf file. Since this problem is difficult to handle, it can still happen that these measures do not take effect — especially in the case of a subsequent setup of prelink.

Therefore, if you install prelink later, set the entry yourself and add the following line to the file with the following command:

root@linux# echo "-c /etc/prelink.conf.d/cmk-update-agent.conf" >> /etc/prelink.conf

Error message cmk-update-agent: error while loading shared libraries: libz.so.1: failed to map segment from shared object

This error message occurs when the /tmp directory with the flag noexec was mounted in the system. To fix this problem, you can either remove the flag, or — if you deliberately set and require the flag — on the Checkmk server in Setup create a rule under Agents > Windows, Linux, Solaris, AIX > Agents > Agent rules > Installation paths for agent files (Linux, UNIX). There you can define the tmp directory in the Directory for storage of temporary data (set TMPDIR environment variable) option yourself. The agent updater plug-in will then in future write temporary files in the defined directory. That works even if you call the plug-in manually with the helper script in /usr/bin/cmk-update-agent.

RPM installation fails on RedHat/CentOS

It has occasionally occurred — especially on RedHat/CentOS systems — that the call to rpm triggered by the automatic update repeatedly fails, while a manual call to cmk-update-agent processes successfully. The cause in these cases was a SELinux policy that prevented an error-free call if rpm was called by a child process of xinetd. You can solve the problem, i.e., get to the bottom of it by analysing the SELinux logs, and adjusting the policy accordingly using the audit2allow tool.

On this page

1. Setting up automatic updates
2. Restricting updates to specific hosts
3. Diagnoses
4. Application scenarios
5. Agent updates in distributed monitoring
- 5.1. Functionality
- 5.2. Configuration
  - Connection to the central Agent Bakery
  - Connection to a remote Agent Bakery
6. HTTPS handling
- 6.1. Using HTTPS
- 6.2. Providing certificates
  - Host to Checkmk server connection
  - Connection from a Remote Site to the Central Site
7. Troubleshooting
- 7.1. Typical errors and their solutions