The commercial editions can update their agents on Linux, Windows and Solaris automatically.
This feature enables easy updating of the agents in the case of new Checkmk versions, and even a changed configuration of the agents can be applied automatically.
In this way you can take advantage of the Agent Bakery to apply individual configurations to hosts.
1. Introduction
This article describes how to set up automatic agent updates on your monitored hosts.
In the section Setting up automatic updates, you will learn what preparatory steps are required and how to enable automatic agent updates for hosts.
The section Skip or disable automatic updates shows you how to specify and test which hosts receive automatic updates.
The section Diagnosis helps you assess whether all updates are being distributed as intended.
The section Deployment scenarios provides tips for implementing the typical scenarios described and highlights additional considerations for each.
The configuration of automatic updates in distributed monitoring is covered separately in the section Agent updates in distributed monitoring.
Since we strongly recommend using HTTPS, you will find specific instructions for securing your setup in the section Working with HTTPS.
The section Troubleshooting lists some common errors, their causes, and possible solutions.
In the section Files and directories, you will find all relevant paths on the Checkmk server and on the monitored host.
2. Setting up automatic updates
To implement the updates, follow these steps: First open Setup > Agents > Windows, Linux, Solaris, AIX and select Agents > Automatic updates:
See Prerequisites for a list of prerequisites that need to be met for the automatic updates to work correctly.
You can simply tick these off in order.
Do not forget that you can get more information for each of these items via Help > Show inline help.
Clicking on 
will take you directly to the setting you need to configure.
Specifically, the settings described in the following five sections must be made and then activated via the Master switch.
2.1. Creating a signature key
The system is designed in a way that the agents can download updates via HTTP or HTTPS from their central monitoring server. Because agents contain executable code it is especially important to guard against the possibility of falsified agents coming from an attacker. Signature keys are used for this purpose. These keys consist of a pair of public and private keys (the public-key method).
You can create as many signature keys as you like. Each of these is secured with a passphrase, which you will subsequently need to enter each time you sign. This prevents, for example, an attacker gaining access to a monitoring server backup from signing agents.
Create a signature key here and record the passphrase.
 |
This signature key can later neither be changed nor restored. If the key is lost, it can mean that all agents need to be updated manually once again. |
2.2. Configuring the update plug-in
The actual update is performed by an agent plug-in named cmk-update-agent.
This is called by the agent in a definable cycle (e.g. once per hour).
At this time it asks the deployment server (your central monitoring system) if there is a new package available for this host, and if so it will then perform the update.
The plug-in must of course be present and correctly configured in the agent. To do this, extend the agents with this plug-in using the Agent updater (Linux, Windows, Solaris) rule set. You can find this rule set by clicking on the Configuration of update plug-in entry on the Automatic agent updates page.
Note that the rule set here follows the ‘First rule per parameter wins’ principle. This allows you to define basic settings in general so that they do not have to be set again and again in the more specific rules. In addition, the inline help provides more information about each item as soon as you activate it.
Below are a few explanations of the individual points.
Activation
Check the Activation option and select the appropriate value (Deploy plugin …) to add the plug-in to the agent. Here, for example, rule inheritance can be used to set a default in a higher-level folder and override this for individual hosts/folders as needed.
Update server information
Optional configuration data for the connection of the Agent Updater to the Checkmk server can be entered here. If this entry is not configured, the information must be entered later when registering the Agent Updater.
Usage
In the case of distributed monitoring with central setup, the Agent Updater receives its relevant update server by default from the Checkmk site to which it connects. This option can be used to configure that the update server entered here is used permanently and that the automatic update server will be ignored.
DNS name or IP address of update server
Here you enter — with a few exceptions — the name of the server on which you are currently configuring this rule. Therefore, you can copy the URL from your current browser window.
An exception to this approach would be if the affected hosts should be ‘moved’ to another Checkmk server. For this situation, once only, enter a different server here. See also below under Deployment scenarios. Enter the DNS name under which the Checkmk server can be accessed. It is important here that the host to be monitored can resolve this DNS name and that it has been configured as a host in Checkmk. When using HTTPS, make sure that the name of the certificate matches the name of the Checkmk server known to the host.
Name of Checkmk site of update server
Here you enter — with a few exceptions — the name of the site on which you are currently configuring this rule. An exception to this approach would be if the affected hosts should be ‘moved’ to another Checkmk site. In such a case, for one time only, enter a different site here. See also below under Deployment scenarios. In a distributed monitoring with central setup, the site where you want to register may also be different from the central site where this rule is configured.
Protocol to use for fetching updates
If — as we recommend — you use HTTPS, you must ensure that the Agent Updater also has a CA certificate (CA = Certification Authority) available to verify the connection.
|
Depending on the server configuration, the use of HTTPS (including forwarding from HTTP to HTTPS) may be forced. In such a case, configuring this rule to HTTP has no effect. |
Certificates for HTTPS verification
CA or self-signed certificates configured here are available to the Agent Updater for the verification of HTTPS connections. Alternatively, in the case of a distributed monitoring with central setup, certificates can also be made available to the Agent Updater from the Checkmk site, or these can be imported directly during the connection via command line parameters.
|
If the server’s certificate chain is signed with a public CA, the connection can normally be verified without imported certificates. However, as soon as imported certificates from one of the mentioned sources are available to the Agent Updater, all other CA certificate authorities are ignored! In the case of problems with the configuration of HTTPS, please consult the following FAQ below. |
Interval for update check
Here you specify the interval in which the Agent Updater queries the configured monitoring server whether any updates are available. As long as the specified interval has not expired, a cached call is returned, so as to burden the network load as little as possible. It usually makes sense to use an interval of not less than 10 minutes, otherwise there is the very great danger that your network will be very heavily burdened if you have a large number of Checkmk agents. If you do not set a value here a default value of 1 hour will apply.
Proxy settings
This rule setting is likewise optional. The Agent Updater initially assumes that there is a direct connection to the Checkmk server on the target host, even with configured proxy settings, and ignores all local proxy settings. If this is the desired behavior this rule setting can therefore be omitted. Otherwise either enter proxy settings manually, or use the host’s existing environment variables.
Executable format (Linux)
You can optionally specify how the plug-in is to be added to the agent installation package. The selection you make here affects only the creation of the updater for Linux hosts.
For 64-bit systems, you can select the packaged binary option. The prerequisite for this option is a x86-64 Linux system with glibc 2.17 or higher. For deployment on 32-bit systems, or in contexts where for other reasons you do not wish to use the binary, select the Python script option instead. The following system requirements apply:
Python version 3.7 or newer
The Python modules cryptography, requests, and pysocks (the last one for SOCKS proxy support)
On hosts running other operating systems, the Agent Updater is deployed in an automatically selected format. For these hosts, your selection for the Executable format (Linux) option is not taken into account.
Signature keys the agent will accept
Select at least one signature key whose signature should be accepted by the Agent Updater. You can also optionally specify multiple keys. This can be the case if, for example, you want to disable an old key. For this purpose the host’s Agent Updater must in the interim accept both keys.
Registration
Starting with Checkmk 2.5.0, it is possible to register the Agent Updater together with the Agent Controller. Using the setting for Registration, you can specify whether and how the joint registration should be performed.
If you want the registration of the Agent Controller to be coupled with the Agent Updater registration, choose between the first two options:
With On first agent controller registration, the first site where you register the Agent Controller will also be used for registering the Agent Updater. In this scenario, you can later register the host on a different Checkmk site without overwriting the Agent Updater’s registration.
With On every agent controller registration, the most recent site on which the Agent Controller was registered is also used for registering the Agent Updater.
If you prefer to always register the Agent Updater separately from the Agent Controller, select the last option, Manual. With this setting, you still have the option to perform the registration of the Agent Updater together with the registration of the Agent Controller on individual hosts.
Generally, registering them together is the more convenient option. Separate registration is also available if, for example, due to changes in the responsibilities of different sites in distributed monitoring, you want to register the Agent Updater individually on a different site without changing the Agent Controller’s registration.
Save all your entries by clicking on 
Save.
2.3. Baking and signing agents
Next, you can immediately bake and sign the agents configured in this way in a single action. This is because the newly created and customized rules will not be found in the installation packages until you have created/baked them again.
To do this, click the Baked agents entry on the Automatic agent updates page, and then click Bake and sign agents. You must now enter the passphrase for the signature key. After you click Bake and sign, the baking procedure will start as a background process. When this process has completed, you will be informed:
All agents signed in this way will be assigned to a corresponding 
symbol.
If you have created multiple keys, sign with these additional keys separately.
It is sufficient for the Agent Updater on the host to be monitored if the new package has been signed with one of the keys known to the updater.
The properties of each available baked agent package appear in the Agent Bakery overview. Here you can check, for example, whether the updater plug-in is included in the agent package (Deploy plug-in that updates agent automatically).
The screenshot shows the left side of the overview table. To the right of it (not shown here) are the icons for downloading this agent package for various target systems. The table also displays the hash for the agent package. This allows you to map agents installed on hosts to entries in the Agent Bakery and thus track which properties were configured for that specific agent.
2.4. Registering the Agent Updater
In the next step register the hosts to be monitored on the Checkmk server. Since a new host is not yet trusted by the Checkmk server, and the server does not yet know that the host should be updated automatically, the agent must be installed manually — once-only — on the host.
To do this, first go to Setup > Agents > Windows, Linux, Solaris, AIX and download the appropriate package for the host from the Agent Bakery. Make sure that the package really includes the Agent Updater plug-in.
Both the Agent Controller and the Agent Updater must each be registered once with your Checkmk site. The steps required to achieve this are explained in the following sections.
The registration process serves to establish a relationship of trust between the host and the Checkmk server. After registration, the Agent Updater on a host may download its own agent from the server at any time without a password. Downloading agents that are meant for other hosts is not possible (as these may contain confidential data).
To ensure that the Checkmk server knows which agent must be provided for which host, the agent and server agree on a secret key (host secret) during registration. This key is stored locally on the Checkmk server, along with the host’s name as it is known in the monitoring system.
This signature ensures the agent’s authenticity regardless of the selected protocol. However, the recommended approach is to use HTTPS, since HTTP does not provide transmission encryption. The agent may contain passwords in scripts and configuration files.
Selecting a user for registration
When registering the Agent Controller and the Agent Updater, you must provide the credentials of a Checkmk user with the appropriate permissions.
The user agent_registration, which is available by default in new Checkmk sites, is suitable for this purpose.
However, you can also specify another user, provided they have the required permissions.
Use the GUI at all is a prerequisite for contacting the deployment endpoint.
-
To register the Agent Controller, the user needs one of the following permissions, depending on whether they are meant to register only their own hosts, or any hosts:
Register any existing host (all hosts)
Register managed existing host (only hosts for which the user is a contact)
-
To register the Agent Updater, the user additionally needs one of the following permissions:
Enable agent updates on all hosts & download all monitoring agents (all hosts)
Enable agent updates on hosts & download monitoring agents (only hosts for which the user is a contact)
Since Checkmk 2.5.0, the agent_registration user has all the permissions required to register the Agent Controller and the Agent Updater.
In addition to the username, you will also need the corresponding Automation secret. For information on configuring the automation user and the corresponding automation password, see the User Guide article on users and permissions.
The automation password is required only once for registration. Following a successful registration, the host authenticates itself to the Checkmk server using the host secret, which was automatically established during registration.
When registering, you must also specify the name of the host as it is known in the monitoring system. Note that this name is not necessarily identical to the hostname of the computer itself.
The following sections demonstrate registration using the agent_registration user.
As an example, the host myhost is registered with the Checkmk server myserver.example.com for the Checkmk site mysite.
When you configured the update plug-in, you already decided whether you want to perform the registration of the Agent Updater simultaneously with the registration of the Agent Controller or handle these two registrations separately. Depending on the selected configuration, proceed now to the section on simultaneous registration, or to the section on separate registration.
Simultaneous registration of the Agent Controller and the Agent Updater
The specific command for simultaneous registration depends on the value you selected for the Registration setting in the Agent updater (Linux, Windows, Solaris) rule. The two options On first agent controller registration and On every agent controller registration ensure that the Agent Updater will be automatically registered when the Agent Controller is registered. The difference between the two options lies in whether the Agent Updater registration is overwritten if you register the Agent Controller with multiple sites.
If you have selected the value Manual, you can still include the Agent Updater registration in the command for registering the Agent Controller on individual hosts.
To do this, include the --automatic-updates option when running the command.
The following examples illustrate a scenario in which simultaneous registration via the Registration option has been configured in the rule Agent updater (Linux, Windows, Solaris).
Therefore, the --automatic-updates option is not required during registration in this case.
Simply execute the command for registering the Agent Controller — the additional registration of the Agent Updater is already included in the agent package and does not need to be explicitly specified when running the command:
- Linux
- Windows
PS C:\Windows\system32> & 'C:\Program Files (x86)\checkmk\service\cmk-agent-ctl.exe' register ^ -s myserver.example.com ^ -i mysite ^ -H mywindowshost ^ -U agent_registration ^ Attempting to register at myserver.example.com, port 8000. Server certificate details: PEM-encoded certificate: -----BEGIN CERTIFICATE----- MIIFijCCA3KgAwIBAgIUaJdZuULIWevTdrW/UjVk8mFWio8wDQYJKoZIhvcNAQEN ... Ayk0aiwc7ylx3cyz9KUyf0fdKaagz2ZLzofuTdjG -----END CERTIFICATE----- Issued by: Site 'mysite' local CA Issued to: mysite Validity: From Wed, 11 Mar 2026 11:56:18 +0000 To Tue, 11 Mar 2036 11:56:18 +0000 Do you want to establish this connection? [Y/n] > Y Password for Checkmk user 'agent_registration': > Registration complete. Attempting to register Agent Updater... Agent Updater registration complete.
The output during registration is similar to the output when registering the Agent Controller without the Agent Updater. The highlighted lines at the end indicate that the Agent Updater registration has also been initiated and completed successfully.
To verify the results of the registration, you can, for example, view the local configuration of the Agent Updater on the host .
You can now enable the distribution of automatic updates in your Checkmk site.
Registering the Agent Updater separately
You can register the Agent Updater separately from the Agent Controller. The prerequisite for this is that a baked and signed agent package with the following properties is available on the host:
It contains the Agent Updater plug-in — after all, you only want to perform the registration manually, not the regular updating of the agent.
It contains the value Manual for the setting Registration in the Agent updater (Linux, Windows, Solaris) rule.
If you use the auto-registration available in Checkmk Ultimate, those agent packages that are configured for it will always include the updater plug-in, with the setting Manual for the updater’s registration.
In the Agent Bakery you can view the properties of a baked agent package to ensure that the above conditions have been met.
For the scenario described here, it is assumed that the host is already known in the monitoring and that the Agent Controller on the host is already registered with your site.
Install the downloaded agent package.
The existing Agent Controller registration remains unchanged, and the Agent Updater plug-in has now also been installed — but the Agent Updater has not yet been registered.
You can verify this by viewing the local configuration of the Agent Updater on the host.
On a Linux host, the file cmk-update-agent.state contains only a single piece of information at this point:
{'last_error': 'The agent updater is not registered at the deployment server'}After installation, you will find the Agent Updater plug-in in the /usr/lib/check_mk_agent/plugins/3600/cmk-update-agent directory.
The 3600 subdirectory corresponds to the update check interval in seconds (here, the default value of one hour).
A script with the same name is also stored under /usr/bin/, so cmk-update-agent is also available as a command.
Use cmk-update-agent --help to get an overview of all available options and modes.
You can now manually register the Agent Updater at any time.
If you enter cmk-update-agent register without any additional arguments, the interactive mode of the Agent Updater will start.
You will be prompted to enter values for various fields that do not yet exist in any configuration file.
You have slightly more control in non-interactive mode, as you can set all fields displayed in the help there.
The values you specify will override those stored in the cmk-update-agent.state file.
However, values from the cmk-update-agent.cfg file are not overwritten.
More details on this can be found below under Viewing the local configuration.
In the scenario prepared here, the cmk-update-agent.state file does not yet contain any useful information for registration.
The code examples below demonstrate the interactive mode, in which you enter all necessary information.
The specific inputs required depend on the information you have already provided in the step Configuring the update plug-in.
- Linux
- Windows
After installing the agent package, the Agent Updater is integrated into the Windows agent at
C:\Program Files (x86)\checkmk\service\check_mk_agent.exe. You can then always issue commands to the updater itself usingcheck_mk_agent.exe updater.The command shown below starts the interactive mode. On Windows, this must be done in a command prompt with administrator privileges.
As an alternative, you can also perform the registration in the non-interactive mode by passing the desired arguments via the command line. You can view an overview of the available options by running
"C:\Program Files (x86)\checkmk\service\check_mk_agent.exe" updater --help.The complete command for registration is as follows:
Once the Agent Updater has been registered, proceed to the Master switch, which allows you to enable automatic update distribution in your site.
2.5. Master switch
From now on, the hosts you have equipped with the agent package and the Agent Updater will check with their assigned Checkmk site at the specified interval to see if agent updates are available.
By default, automatic agent distribution is initially disabled globally in your site.
To finalize the setup, activate it by clicking Master switch.
This changes the Activation of automatic agent updates global setting under Setup > General > Global Settings > Automatic Agent Updates.
All conditions should should now have been fulfilled. The Prerequisites table should now look like this:
From now on, the Agent Updater will regularly check for a new version of the agent, following the configured update interval. As soon as a new version is available and signed, the Updater will automatically download and install it.
At the same time, the Master switch provides a way to globally disable updates.
To verify that updates are being distributed as intended, you can now jump to the section on diagnosis.
3. Skip or disable automatic updates
Once set up, at the configured interval the Agent Updater will download and install any new compatible agents from the Agent Bakery. If you want to temporarily or permanently disable the automatic distribution of updates for one, several, or all hosts, you have the following options.
3.1. Completely remove the Agent Updater plug-in from a host
To completely remove the Agent Updater from a specific host, use the Agent updater (Linux, Windows, Solaris) rule set to adjust the host’s settings so that the update plug-in is disabled there. The agent will then remove its updater automatically with the next scheduled update!
It goes without saying that the Updater can only be reactivated by manually installing a new agent package. However, the registration will remain unchanged and does not need to be renewed.
3.2. Restricting the distribution of updates to specific hosts
To restrict the distribution of agent updates to specific hosts in your site, you can configure distribution conditions. This can be useful, for example, to test a new agent on a small number of hosts first before rolling it out to a larger number of hosts. This important step prevents a potential error from becoming more widespread.
The middle box on the Automatic agent updates page is used for this purpose:
The conditions are always linked by a conjunction (logical AND). Only hosts that meet all of the specified conditions will receive the update. Once you have defined conditions for selecting hosts here, you can enter individual host names in the Test hostname before activation field and check whether updates are now enabled for these hosts or not.
You can also access the page for configuring the conditions via Setup > Global settings > Automatic agent updates > Selection of hosts to activate agent updates for.
To clear all conditions and thereby reactivate the distribution of updates to all hosts, use the Reset to default option on the Setup > Global settings > Automatic agent updates > Selection of hosts to activate agent updates for page.
3.3. Disabling the master switch
To stop the distribution of agent updates entirely within your site, simply disable the Master switch. All settings you have configured in the instance and on the hosts will be preserved.
You can re-enable the Master switch at any time.
4. Diagnosis
There are quite a few sources of information for diagnosing whether all updates work as intended.
4.1. Agent deployment statistics
This overview on the Automatic agent updates page shows how individual hosts behave during agent updates.
The inline help gives further explanations.
Clicking on 
provides a detailed list of the individual hosts.
You can also get to the complete list of all registered hosts via the Monitor > System > Agent update status view.
There you can then search for individual hosts.
In this list you will also find documentation on how the hash of an agent starts, which agent is intended for a host (Target Agent), which agent was most recently downloaded from the host (Downloaded Agent), and which agent is currently installed on the host (Installed Agent). In this way you can always see if the specifications have been met or where the process is currently located.
The status information shown in the screenshot is acquired directly during communication between the Agent Bakery and the Agent Updater.
To the right of the displayed section, there are also the fields Update Check and Update Check Output.
Since Checkmk 2.5.0, the values in these fields are provided by a dedicated status plug-in (cmk-update-agent-status on Linux/Solaris, cmk-update-agent-status.ps1 on Windows) on every request by the agent, and thus always correctly represent the current state.
4.2. The Check_MK Agent service
If you have installed the Agent Updater plug-in on an agent, it will regularly output the current status of the update in the form of monitoring data. The service discovery generates a new service with the name Check_MK Agent on the host. This reflects the current state of the update. This way you can be notified of any problem with the updates.
The service also indicates the status of the signature keys that the Agent Updater trusts. However, this information is only presented as a note in the service details, and does not trigger a WARN or CRIT state at the host level. Because the signature keys are managed centrally in the Agent Bakery, warnings for keys that are about to expire are also managed centrally and not on the level of individual hosts.
4.3. Viewing the local configuration
The behavior of the Agent Updater is governed by the two files cmk-update-agent.cfg and cmk-update-agent.state.
It always applies that defined values from the .cfg file override those from the .state file.
Since Checkmk 2.5.0, the Agent Updater does not run as root anymore on Linux, but instead under the Agent Controller user (cmk-agent).
Correspondingly, the status and log files of the Agent Updater are found in dedicated directories owned by this user.
If the Agent Updater shows unexpected behavior, it is sometimes worth having a look at the configuration. There is also a handy feature if you call the Agent Updater directly from the command line:
In Windows, the corresponding command is 'C:\Program Files (x86)\checkmk\service\check_mk_agent.exe' updater show-config.
4.4. Log messages
In the event of a problem you will also find log data for the updates on the host to be monitored.
On Linux cmk-update-agent logs important information to syslog — such as warnings and errors:
Jul 02 13:59:23 myhost [cmk-update-agent] WARNING: Missing config file at ./cmk-update-agent.cfg. Configuration may be incomplete.
Jul 02 13:59:23 myhost [cmk-update-agent] ERROR: Not yet registered at deployment server. Please run 'cmk-update-agent register' first.A more detailed log file cmk-update-agent.log including debug output and possible tracebacks can be found on Linux and on Windows:
2026-01-16 07:49:23,295 [288229] DEBUG: Starting Checkmk Agent Updater v2.4.0p19
2026-01-16 07:49:23,295 [288229] DEBUG: Updating the certificate store "/var/lib/check_mk_agent/cmk-update-agent/cas/all_certs.pem"...
2026-01-16 07:49:23,297 [288229] INFO: Updated the certificate store "/var/lib/check_mk_agent/cmk-update-agent/cas/all_certs.pem" with 1 certificate(s)
...
2026-01-16 07:49:23,326 [288229] DEBUG: Successfully read /var/lib/check_mk_agent/cmk-update-agent/cmk-update-agent.state.
2026-01-16 07:49:23,326 [288229] DEBUG: Successfully read /var/lib/check_mk_agent/cmk-update-agent/cmk-update-agent.state.
2026-01-16 07:49:23,327 [288229] DEBUG: Saved deployment status to /var/lib/check_mk_agent/cmk-update-agent/cmk-update-agent.state.
2026-01-16 07:49:23,327 [288229] INFO: Target state (from deployment server):
2026-01-16 07:49:23,327 [288229] INFO: Agent available: True
2026-01-16 07:49:23,327 [288229] INFO: Signatures: 1
2026-01-16 07:49:23,327 [288229] INFO: Target hash: 37221b87f5cb46a2
2026-01-16 07:49:23,327 [288229] INFO: Agent 37221b87f5cb46a2 already installed.
2026-01-16 07:49:23,340 [288229] DEBUG: Caught Exception:
Traceback (most recent call last):
File "cmk_update_agent.py", line 1733, in main
File "cmk_update_agent.py", line 714, in run
File "cmk_update_agent.py", line 1372, in _run_mode
File "cmk_update_agent.py", line 1071, in _do_update_as_command
File "cmk_update_agent.py", line 1150, in _do_update_agent
File "cmk_update_agent.py", line 1221, in _check_signatures
Exception: No valid signature found.Under both systems you can also use the command line option --logfile LOGFILE to specify an alternate path for the log file.
5. Deployment scenarios
5.1. Migrating to a new monitoring site
Should you want to move to a new Checkmk site in a single-site setup without losing the hosts registered on the server, it should be noted that for a successful agent update process the following information on server and host must match:
The name under which the host is monitored and registered.
The host secret, which was automatically assigned during registration.
The signature used to sign the agents.
To achieve this, follow these steps:
First add all hosts whose registration information is to be migrated to the new site to the monitoring. Make sure the hosts in the new site are monitored under the same name. Then copy the
~/var/check_mk/agent_deploymentfolder from the old to the new monitoring site.Export the signature key(s) that are accepted by the agents installed on the hosts. The signature keys can be exported and imported using Setup > Agents > Windows, Linux, Solaris, AIX > Agents > Signature keys.
Import these signature keys from step 2 on the new monitoring site.
Configure the Agent Updater rule on the new monitoring site according to the instructions, and sign the baked agents with the imported signature key(s).
Lastly, in the Agent Updater rule on the old site, configure the fields for the update server and the name of the Checkmk site conforming to your new monitoring site, and bake the agents again. Note: Please check at this point that you have specified everything correctly before you re-bake the agents.
As soon as the next automatic updates have been processed on the hosts, the Agent Bakery of the old monitoring site will be locked out. From that time on the hosts to be monitored will only answer to the new Checkmk server. Following the second automatic update the agent will be installed by the new Checkmk server accordingly.
5.2. The Agent Updater as automatic installer
|
This is not an official feature of the Agent Updater. These instructions are therefore intended primarily for more experienced users. The official way to install a Checkmk Agent on a host is to download and run the agent package appropriate for the system. It is however also possible to allow the Checkmk Agent to be installed initially by the Agent Updater, since this also works as a stand-alone program. |
Proceed as follows:
Copy the cmk-update-agent binary or the
cmk_update_agent.pyscript to the host to be monitored (both can be found at~/share/check_mk/agents/pluginson the Checkmk server).Register the host on the Checkmk server by invoking
cmk-update-agent register. Here it makes sense to pass the required registration information directly via the command line – especially if you want to use an installation script. The corresponding options can be displayed when callingcmk-update-agent register --help.Then, with a final call to the Agent Updater plug-in, install the agent with all of the configuration details for the host being monitored. However since there is no local configuration (the Agent Updater also displays a corresponding warning), and thus no signature for the agent package to be downloaded, call the updater once more with
cmk-update-agent --skip-signaturesto explicitly trust the downloaded package. The prerequisite for the installation by Agent Updater is, of course, that the Agent Bakery has a suitable agent package ready for the target host on the Checkmk server.
6. Agent updates in distributed monitoring
Since Checkmk 2.0.0 it has also been possible to distribute baked agents via remote sites. The prerequisite for this is a distributed monitoring with central setup and a connection which allows the central site to be reached from the remote sites via HTTP/HTTPS.
Such a distributed monitoring can — especially temporarily — also be operated with different Checkmk versions. This makes it possible to successively switch larger systems to a new Checkmk version. In this case, however, be sure to follow the notes on mixed-version monitoring environments in distributed monitoring.
6.1. Functionality
Technically, the feature is implemented in such a way that requests for updates on remote sites are forwarded to the central site — so that the entire configuration as well as the baking of the agents takes place exclusively on the central site. Agent packages that have already been requested at a remote site are cached on the remote site (as long as they are valid), so that the packages do not have to be downloaded again from the central site. In addition, the requested data will be checked for consistency on the remote site, so that unnecessary connections to the central site are avoided.
Unlike in a single-site setup, the appropriate update server for the hosts does not originate exclusively from the set of rules for the Agent Updater, but is communicated to the requesting Agent Updater by the contacted Checkmk site. In this process, a host is provided with its server by the site from which it is being monitored. The specification of a Checkmk server is therefore only needed for the one-time registration, which can theoretically take place at any site — that is accessible from the host — in the entire distributed monitoring system. If the connection to the automatically determined server fails, the previously saved server (from the rule configuration or previously-entered manually during registration) is used as a fallback.
6.2. Configuration
The distribution of agent packages via remote sites does not have to be activated separately — the respective remote site recognizes the situation automatically and accordingly communicates with the central site as well as the requesting Agent Updater. If the agents are to report explicitly only to the central site for updates, this can be done via a fixed update server in the rule set for the Agent Updater.
However, to ensure that agent updates actually work in distributed monitoring, a few settings must still be configured on the central site — these are all located in Setup > General > Global settings in the section titled Automatic agent updates.
If the settings differ for each remote site, they can be edited separately as well.
To do so, go to Setup > General > Distributed monitoring and choose the gear icon 
in the desired site’s row to access that site’s Site specific global settings.
Connection to central agent bakery
At this point, the URL through which the central site can be accessed from the remote site must be specified, including its protocol and the character string /check_mk/, for example https://myserver.example.com/mysite/check_mk/.
While the Checkmk site will try to identify all other missing settings itself, the specification of this URL is not optional, as this connection direction will otherwise not be established in the case of a central configuration.
If the protocol is HTTPS, the remote site automatically uses the CA or self-signed certificates available in the setup for the verification of the connection Setup > General > Global Settings > Site management > Trusted certificate authorities for SSL.
Connection to remote agent bakery
Since the remote sites are not necessarily accessible from the respective monitored hosts via the same URL as from the central site, a URL can be configured here for this purpose. This URL is then automatically communicated to the host (or the requesting Agent Updater) when a connection is made to a Checkmk site. The configuration as Site specific global settings makes particular sense here. If no URL is specified, it is assumed that the remote sites are accessible from both the central site and from the monitored hosts under the identical URL. If it is an HTTPS connection, the appropriate certificate can also be automatically made available to the host. Since the central CA store cannot be used for this, appropriate certificates can be specified at this point. Alternatively, the certificates can also be specified in the Agent Updater rules.
7. Working with HTTPS
At various points in this article there are references to securing the respective connections via HTTPS. Here, we will once again provide a general overview of what needs to be done to fully secure connections via HTTPS. Both the connection from a remote to its central site, and from a host to a Checkmk site can and should be secured via TLS, i.e. using HTTPS. This is independent of whether it is a single site setup or a distributed monitoring.
7.1. Configuring HTTPS
In order to be able to connect to a Checkmk site via HTTPS, first the monitoring server must be configured for HTTPS. This can be achieved, for example, via a suitable configuration of the system Apache, or most simply via the HTTPS settings on the Checkmk appliance.
Whether a Checkmk server is then addressed via HTTP or HTTPS is determined by the URL configured in each case.
If this begins with https://, the server is addressed via the HTTPS protocol using port 443.
This also applies to the protocol you configured with the Update server information setting.
In principle, you can force forwarding from HTTP to HTTPS on the Apache side and (initially) exclude individual paths for this.
For configuration details, see the Apache documentation for the mod_rewrite and mod_redirect modules.
7.2. Providing certificates
In order for an HTTPS connection to be established, it must be possible to verify the contacted server’s certificate chain or the self-signed certificate (depending on how the server is configured). The provision of suitable CA or self-signed certificates makes this possible and can be realized in various ways.
Connections from a host to a Checkmk server
The Agent Updater always attempts to verify HTTPS connections and terminates these if a verification is not possible. Certificates for verification are available to the Agent Updater from the following sources:
The Agent Updater: By default, the Agent Updater comes with a Python runtime environment including any required modules. Also included is the certificate bundle of the Certifi module, which in turn is based on the Mozilla project’s certificate collection. This ensures that public CAs are made known to the Agent Updater in a timely manner, even when operating system updates are pending.
Certificates via the Agent Bakery: Certificates contained in the Certifi module are ignored once one or more certificates have been imported via one of the following mechanisms. Certificates from the following sources are stored locally on the host and used only by the Agent Updater:
Using the Certificates for HTTPS verification setting, certificates can be baked into an agent package and will be available to the Agent Updater from installation (or an update).
When configuring the connection to the remote site via Setup > General > Global settings > Automatic agent updates > Connection to remote agent bakery, you can specify certificates that can be used to verify the HTTPS connection to the respective remote site. This is especially useful if it is not yet clear at configuration time which host will be assigned to which site. This import option can also be used to reduce the number of agents to be baked, since the correct certificates for the respective update servers do not have to be host-specifically configured.
Certificate Store:
the Agent Updater may use the operating system’s Certificate Store only if the Agent Updater has been configured to use the System-Python instead of the supplied Python runtime environment, and no Certifi module is configured in the System-Python.
Since many factors such as the installation parameters or the _PIP_STANDALONE_CERT environment variable are involved here, we do not officially support such a configuration.
Certificate via command line - Import certificate:
You can also call the Agent Updater with the command line argument --trust-cert.
This retrieves and imports the server certificate.
This is done without taking into account the type of certificate:
The certificate is trusted without further checking of a possible chain - regardless of whether it is a self-signed certificate, a certificate at the end of a public certificate chain or a certificate signed with an internal CA.
|
|
Certificate via command line - Ignore certificate:
If there is no valid certificate available to the Agent Updater at all, the certificate validation can be bypassed by using the --insecure command line argument for a call.
This can be useful if the valid certificate is already waiting to be retrieved from the server on the next connection,
but the Agent Updater will be 'locked out' by this missing certificate.
|
This actually completely deactivates the certificate check for this call. Communication still takes place in encrypted form, so the use of this argument is 'better than nothing'. |
Connection from a remote site to a central site
The distribution of certificates is simpler when connecting from a remote to the central site, since the setup procedure is not exited at all. The remote site can use the Checkmk certificate store under Global settings > Site management > Trusted certificate authorities for SSL. It is therefore sufficient to import the certificate(s) via the central site, if necessary also via Site specific global settings when the central site is accessible under multiple URLs.
Procedure for replacing a certificate
If you work with your own certification infrastructure, you ideally use a root certificate that is valid for a very long time and whose associated key is regularly used to create intermediate certificates. These are then in turn used to sign the server certificates. In this case, you roll out intermediate certificates as a certificate chain on the Checkmk server. The hosts that receive automatic agent updates now only need to be made aware of the root certificate.
If you are unsure whether a new server certificate requires a new root certificate, use the following command. Use it to determine the identifier of the root key used to sign a server certificate:
If the displayed identifier is identical for both the old and the new certificate, no further action is required. Hosts that trust this root certificate can continue to obtain updates even if the certificate chain changes — provided that the chain was correctly stored in the system Apache.
If a self-signed certificate or a short-lived root certificate from an internal CA has been used up to now, or if the previous root key of one of your internal CAs has been compromised, proceed as follows when replacing the certificate:
Add the new certificate using the Certificates for HTTPS verification parameter (in the Agent updater (Linux, Windows, Solaris) rule).
Re-bake the agent packages and update all hosts in the monitoring. Ensure that this update has been run for all hosts before proceeding.
Now replace the server certificate.
Test with a few hosts in which it is easy to reinstall the agent manually if it fails, to see if it is possible to update again using the new certificate.
If the preceding step (4) could be performed successfully — you can (the certificate will expire), or must (the key was compromised) — remove the old certificate and perform another agent update.
8. Troubleshooting
This chapter first deals with typical, general errors and their causes:
The Agent Updater will really only be run once within the update interval, so an error will be continuously-displayed until either you call the plug-in manually, or the next interval is pending.
The Agent Updater creates its own status file cmk-update-agent.state independently (under Linux/Unix in /var/lib/check_mk_agent/cmk-update-agent/, and under Windows in the config folder).
This file remains on the host after uninstallation, so that the registry information does not get lost.
A new installation will find the file and continue using it.
If this situation is undesirable, simply delete the cmk-update-agent.state file manually after an uninstall.
The Monitor > System > Agent Update Status page displays all of the hosts that are are in the monitoring and for which a status file exists on the Checkmk server.
It does not matter if the host actually reports to the Checkmk server for automatic updates.
Should an unexpected host be displayed here, it is worth taking a look in the /omd/sites/mysite/var/check_mk/agent_deployment folder — the cause will probably be an old or accidentally-created registry.
The Agent Updater is designed to explicitly trust only the certificates which are usually specified under Agent updater (Linux, Windows, Solaris) in the HTTPS configuration. In particular locally-installed certificates are ignored. It can also occur that the Checkmk server is accessible through the browser, while the Agent Updater cannot connect due to an incorrect configuration.
In the HTTPS configuration of the Agent Updater rule a root certificate must be specified with which the connection to the Checkmk server can be verified. In other words: the certificate chain included in the Checkmk server’s server certificate must be verifiable by the certificate given here. Often the server certificate is specified here instead — this is however not suitable for this purpose.
Take a look at the Checkmk server’s certificate chain with the OpenSSL tool.
Due to the chain’s length, for clarity here only a relevant section is shown and the omitted sections marked by [...]:
For the last entry — in our case subject=/CN=mymonitoring.example.net — you need a valid root certificate.
This must not necessarily be — as in this example — the issuer of the certificate.
It will usually be a chain of issuers.
Then look at the certificate used. Here too due to the length it will be shortened as in the above example:
The top certificate — seen in the above excerpt — is not permitted to have a dependency on another certificate.
You can see that the Issuer and the Subject are identical and that the option CA:TRUE is included.
In addition the issuer’s chain that authenticates an object must be consistent up until the final entry.
You therefore also need all intermediate certificates if the issuer of the last certificate should not be a CA.
It has occasionally occurred — especially on Red Hat/CentOS systems — that the call to rpm triggered by the automatic update repeatedly fails,
while a manual call to cmk-update-agent processes successfully.
The cause in these cases was a SELinux policy that prevented an error-free call if rpm was called by a child process of xinetd.
You can solve the problem, i.e., get to the bottom of it by analyzing the SELinux logs, and adjusting the policy accordingly using the audit2allow tool.
The following section deals with problems that are indicated by specific error messages:
On some Linux systems the program Prelink is installed and a cronjob is activated
which regularly examines all binary files on the system, and adapts them if necessary to speed up the programs.
However the Agent Updater plug-in is packaged with the PyInstaller program which is not compatible with such actions, and is therefore broken.
Checkmk therefore has a blacklist entry for deb/rpm packages which is stored under /etc/prelink.conf.d,
and — if prelink exists — sets an entry in the existing /etc/prelink.conf file.
Since this problem is difficult to handle, it can still happen that these measures do not take effect — especially in the case of a subsequent setup of prelink.
Therefore, if you install prelink later, set the entry yourself and add the following line to the file with the following command:
This error message occurs when the /tmp directory with the flag noexec was mounted in the system.
To fix this problem, you can either remove the flag, or — if you deliberately set and require the flag — on the Checkmk server in Setup create a rule under Agents > Windows, Linux, Solaris, AIX > Agents > Agent rules > Installation paths for agent files (Linux, Unix).
There you can define the tmp directory in the Directory for storage of temporary data (set TMPDIR environment variable) option yourself.
The Agent Updater plug-in will then in future write temporary files in the defined directory.
That works even if you call the plug-in manually with the helper script in /usr/bin/cmk-update-agent.
If the Check_MK Agent service displays the warning No valid signature found, this means that the agent package intended for the host in the Agent Bakery has not been signed with one of the accepted keys.
In the simplest case, all you need to do is sign your agents using the Sign agents function in the Agent Bakery with one of the keys displayed under Signature keys the agent will accept.
As soon as the Agent Updater makes its next report from the affected host to the Checkmk server, and the cache interval for the service has expired, the warning will disappear.
However, if the host does not have (or no longer has) a single signature key that is located on the Checkmk server, you must repeat the bake and sign with a key displayed under Signature keys the agent will accept, then copy the baked agent to the affected host and reinstall it there.
On an affected host, you can run cmk-update-agent -v or check_mk_agent.exe updater -v to get more details about this error.
The detailed error message explicitly lists the signatures from the Agent Updater plug-in’s configuration file that do not have an accepted (i.e. configured) counterpart on your Checkmk server.
Ignoring signature #1 for certificate: certificate is unknown.
No valid signature found.9. Files and directories
9.1. File paths on the Checkmk server
| File path | Description |
|---|---|
|
Contains the baked agents, sorted first in subdirectories by operating system (e.g. |
|
Contains files with the names of the registered hosts. One such file contains the time of the last registration and the host secret. |
9.2. File paths on the monitored host
- Linux
File path Description /usr/lib/check_mk_agent/plugins/3600/cmk-update-agentThe Agent Updater plug-in as binary or script, depending on the configuration in executable format. The subdirectory
3600stands for the interval for the update check in seconds (here for the default value of one hour)./usr/bin/cmk-update-agentScript to call the Agent Updater plug-in and register the agent with the command
cmk-update-agent register./etc/check_mk/cmk-update-agent.cfgConfiguration file for the Agent Updater plug-in, which contains the settings of the Agent updater (Linux, Windows, Solaris) rule. Do not edit this file, because it is written to during installations and updates.
/var/lib/check_mk_agent/cmk-update-agent/cmk-update-agent.stateFile with registry information, including the host secret.
/var/lib/check_mk_agent/log/cmk-update-agent/cmk-update-agent.logExtensive log file with debug messages.
- Windows
File path Description C:\ProgramData\checkmk\agent\plugins\cmk_update_agent.checkmk.pyThe Agent Updater plug-in as a Python file.
C:\Program Files (x86)\checkmk\service\check_mk_agent.exeAgent program to register the agent with the command
check_mk_agent.exe updater register.C:\ProgramData\checkmk\agent\config\cmk-update-agent.cfgConfiguration file for the Agent Updater plug-in, which contains the settings of the Agent updater (Linux, Windows, Solaris) rule. Do not edit this file, because it is written to during installations and updates.
C:\ProgramData\checkmk\agent\config\cmk-update-agent.stateFile with registry information including the host secret.
C:\ProgramData\checkmk\agent\log\cmk-update-agent.logExtensive log file with debug messages.
C:\ProgramData\checkmk\agent\bakery\check_mk.bakery.ymlConfiguration file created by the Agent Bakery, which among other things defines the interval for the update check.