Checkmk
to checkmk.com

1. Introduction

In this article you will find the most important topics relevant to an update of your Checkmk from version 2.3.0 to 2.4.0.

We recommend that you read through the entire article before updating so that you know exactly what to expect: before, during and after the update.

Tip

This article will continue to be revised after the release of Checkmk 2.4.0. If an important topic is missing, please report an issue via GitHub. The content on this page has been partly automatically translated and may not always up to date with the German version.

2. Preparations

This chapter gives you an overview of the topics you should take into consideration before you perform the update. Probably not every topic will be relevant for you: For each topic, you can check the corresponding box for your own reference and immediately move on to the next topic.

2.1. Backup

As with any update to production software, you should check that your backups are up to date before performing the update of Checkmk.

Does this affect you? Yes.

What will you need to do? If you create your backups automatically via Setup > Maintenance > Backups, check there that your backups are up to date and that the most recent backup jobs have completed without errors.

For more information, see the articles on Backups and on Backing up and restoring sites.

Tip

From the update to 2.3.0, changes are rolled back in the event of a failed update (Werk #16408). An update can fail due to an unexpected internal error, but also due to user input during the update process, for example by selecting the menu option abort or by pressing the key combination CTRL+C. Only when the text Completed verifying site configuration. Your site now has version 2.4.0pX is displayed, the updated configuration is active. Restoring the configuration does not replace a backup, but usually reduces downtime if something goes wrong.

2.2. Check hardware utilization

Checkmk 2.4.0 requires slightly higher system resources than 2.3.0. It is therefore advisable to find out before updating which free capacity the servers used for Checkmk still have.

Does this affect you? Yes, but the extent to which you are affected depends heavily on how you use Checkmk.

What will you need to do? Make sure that there is sufficient free capacity available. A slightly higher processor load is to be expected when using many active checks. Other areas could be optimized in return, which in many cases leads to a reduction in the processor load. As a rule of thumb: If the CPU load is below the number of processor cores × 0.8 and the CPU utilization is below 70 %, no problems are to be expected.

The memory usage of Checkmk 2.4.0 is about 30 % higher compared to 2.3.0. The reason for this is more extensive caching of frequently required data, which leads to lower latencies and higher performance at the expense of memory usage.

When using piggyback in distributed monitoring, the RabbitMQ message broker requires additional RAM and CPU resources. The same applies to the Open Telemetry Receiver.

The hard disk space requirement increases by up to 5 MB per host with Checkmk 2.4.0. This additional disk space will be occupied immediately after the update.

2.3. Linux distribution versions

Some obsolete distributions will no longer be supported under Checkmk version 2.4.0.

Does this affect you? This will affect you if any of the following Linux distributions — still supported in 2.3.0 — is installed on your Checkmk server:

  • Debian 10

  • SLES 12 SP5

  • Ubuntu 20.04

What will you need to do? Before updating Checkmk, first perform a Linux distribution version upgrade. Make sure that the target version of the Linux distribution of Checkmk is 2.3.0 and that 2.4.0 is supported. You can find out which Linux distribution versions Checkmk supports in the update matrix for 2.4.0 and on the download page after you have selected the Checkmk version and your Linux distribution.

Shortly after the release of Red Hat Enterprise Linux 10, we will provide Checkmk 2.3.0 and 2.4.0 packages for this distribution version. This will allow you to first update to the latest patch version of Checkmk 2.3.0, then update from Red Hat Enterprise Linux 9 to 10 and finally update to the latest patch version of Checkmk 2.4.0.

2.4. Browser support

Checkmk 2.4.0 uses new JavaScript features that are not available in older browsers. Which browsers are supported in which versions can be found in the Release notes.

Does this affect you? You will usually have automatic updates to the latest version enabled on desktop systems.

What will you need to do? Check the browser version you are using and install a more recent browser if necessary. If you are using single board computers, smart TVs or digital signage solutions to display dashboards and you have no control over the system browser, test that any required dashboards are displayed correctly before updating. If necessary, contact the hardware manufacturer for updates.

Tip

In case you are using Firefox ESR, wait with the update of Checkmk until you have updated Firefox from version 128 to 140. Firefox ESR 140 will be available from the end of June and will be the only supported ESR version as of the end of September.

2.5. Authentication by GET parameter

Up to Checkmk 2.2.0, every automation user could be logged in with a user name and password passed via GET parameters. This option has been reduced as of Checkmk 2.3.0 and will be completely removed in version 2.5.0 (Werk #16223). In the medium term, you must therefore switch your scripts to authentication via HTTP headers.

First, 2.3.0 made the option of logging in via the GET parameter configurable. However, the global setting Enable automation user authentication via HTTP parameters initially used on as the default. In Checkmk 2.4.0 default now changes to off.

Does this affect you? This change also primarily affects users who use single board computers, smart TVs or digital signage solutions without a connected keyboard to display dashboards.

What will you need to do? In case you use automatic login via GET parameters, you must change the global setting Automation user authentication via HTTP parameters to on. As the option of logging in via GET parameters will be completely removed with Checkmk 2.5.0, you should switch the login of such dashboards to Basic Auth (for example, via a browser extension).

2.6. ntopng 5.6 and older are no longer supported

Checkmk 2.3.0 is the last version that supports ntopng in versions 5.x and 6.x (Werk #16483). With Checkmk 2.4.0, support for ntopng 5.x is discontinued and only ntopng 6.x will be supported.

Does this affect you? This change affects users of the commercial editions that use the integration with ntopng.

What will you need to do? Update your ntopng installations to 6.x before updating Checkmk to 2.4.0.

2.7. Changes to the execution of special agents

If special agents are configured for a host, in Checkmk 2.4.0 all these special agents will be executed. In earlier versions only one special agent was executed, namely the first one in the alphabetical sorting.

Does this affect you? This change is only relevant for hosts in whose properties the Checkmk agent / API integrations attribute is set to API integrations if configured, else Checkmk agent.

It is possible that you have inadvertently created rules that have assigned more special agents to hosts than required. These additional rules were previously ignored if they were alphabetically after the first one. Example: You have created rules for agent_acme and rules for agent_cyberdyne and assigned these rules to the same hosts due to a small oversight. In Checkmk 2.4.0 not only agent_acme but also agent_cyberdyne will now be executed for these hosts.

What will you need to do? When using special agents that access APIs that can cause costs or run into rate limiting, you must take preparatory action and ensure correct assignment. Also ensure correct assignment for special agents that cause a high CPU load or a lot of network traffic. Furthermore, bear in mind that if special agents are incorrectly assigned to hosts, services may suddenly appear that do not belong there.

2.8. Uninstalling Python modules

Checkmk 2.4.0 updates Python from to 3.12.3. This update affects modules installed to a Checkmk site. In many cases, post-installed modules are incompatible with Python 3.12.3. In the worst case, obsolete modules overwrite the functionality of the modules supplied by Checkmk.

Does this affect you? This will only affect you when you have explicitly-installed Python modules for special agents or agent-based check plug-ins that you have written yourself or obtained from the Exchange. If you are unsure, carry out the check described in the following section.

However, this also affects you if you use the active check check_sql to monitor Oracle databases. Up to Checkmk 2.3.0p27 you had to install the oracledb module in order to be able to use this check. Since 2.3.0p28 this module is supplied with Checkmk (Werk #17370).

What will you need to do? First find out whether Python modules are installed in the site, and if so — identify them. To do this, search for the dist-info and egg-info directories:

OMD[mysite]:~$ find ~/local/lib/python3/ -type d -name '*.*-info'
local/lib/python3/cryptography-41.0.5.dist-info
local/lib/python3/ecdsa-0.18.0.dist-info

Make a note of the installed modules and then uninstall them:

OMD[mysite]:~$ pip3 uninstall cryptography ecdsa

You can find out how to deal with uninstalled Python modules after the update below.

2.9. Packaged (MKPs) and non-packaged local files

Local files allow you to customize and extend the functionality provided by Checkmk. These files are located in the local part of the site directory structure, i.e. in ~/local and can be located in extension packages or simply 'lying around'. Local files can cause problems during an update if they no longer match the new Checkmk version.

Does this affect you? Since it is not possible for Checkmk to fully ensure the compatibility of local adaptations during an update, you should check your Checkmk site before an update to see whether local files are used, which ones they are and how they are organized.

What will you need to do?

  1. Use mkp list to create an overview of existing extension packages (MKPs) and their sources: You can usually easily test the extensions you have developed yourself and adapt them if necessary. For externally sourced MKPs, you should research whether there are known problems with Checkmk 2.3.0 and whether new versions are available. In cases where functionality previously provided by the MKP is now provided by Checkmk 2.3.0, deactivate the package before updating.

  2. Get an overview of unpackaged local files of your Checkmk site by executing the command mkp find as site user. If paths with python3 appear here, go back to Python modules again. The following applies to all other files: Combine files that belong together into MKPs. This makes it easier to later deactivate these en bloc if incompatibilities are detected following the update.

  3. In the case of MKPs that require different versions for Checkmk 2.2.0 and 2.3.0, you should have installed both package versions with correct compatibility information in Checkmk before performing the update. If different versions of a package are available, Checkmk automatically activates the appropriate version during the update. As central sites with Checkmk 2.2.0 can hold packages for 2.3.0 and distribute these to remote sites, this feature also helps when updating in a distributed monitoring with central setup.

Tip

If you have activated the MKP for the Redfish special agent supplied in version 2.3.0, deactivate it before the update. The presence of the package causes error messages during the update (status 2.4.0b5).

2.10. Programming interfaces

In Checkmk 2.3.0 some programming interfaces (APIs) that were defined ad-hoc in older versions already were replaced by versioned, well-specified ones. The older APIs could continue to be used in parallel. With Checkmk 2.4.0 no further measures are taken to maintain compatibility with the old APIs (Werk #17201). This means that plug-ins that use the old APIs will no longer work with Checkmk 2.4.0.

Does this affect you? The topic of APIs affects you if you have extended the check plug-ins delivered with Checkmk with your own, self-written plug-ins and/or if you use plug-ins from other sources.

What will you need to do? Before updating to 2.4.0, first make sure that you have updated to the latest patch version of 2.3.0. Existing extensions that will no longer work after an update to 2.4.0 will then lead to messages being displayed in the GUI repeatedly for all users with the permission to manage MKPs (Werk #17649).

Review extensions that you have written yourself and those from third-party providers for the use of the current APIs (cmk.agent_based.v2, cmk.server_side_calls.v1, cmk.graphing.v1 and cmk.rulesets.v1). If the older, non-versioned APIs or cmk.agent_based.v1 are used, migrate them to the new APIs and the new folder structure. You can find information on migration in Werk #16259 and in the articles on developing check plug-ins. We also provide scripts that can help with the migration.

Tip

If you use non-public APIs in plugins, test all the more intensively with 2.3.0 and 2.4.0. Some changes have been made to the 'internal toolbox', for example effects on the detection of the debug flag.

2.11. Deprecations in the REST API

Deprecated endpoints or parameters of the Checkmk REST API are marked in the API documentation with the icon.

Does this affect you? This may affect you if you use the Checkmk REST API in your own scripts, for example.

What will you need to do? Open the REST API documentation with Help > Developer Resources > REST API documentation. Search for Deprecated in the browser on the page, check whether you are using deprecated elements in your scripts and replace them before updating. In the REST API documentation, you will usually find instructions on how to replace the functions that no longer exist in the new version.

2.12. Incompatible changes

As in every Checkmk version, there are also changes to the software in the current 2.4.0 version which may have repercussions on your Checkmk installation — at Checkmk these are organized and documented in what we refer to as our Checkmk Werks. A so-called incompatible change requires that you make manual modifications in order to allow existing functions to continue to run as usual and/or to be able to use new functions.

Does this affect you? In general, there will be incompatible changes that will also affect your Checkmk installation, it is however impossible to make a general statement. In this article we have collected those issues that apply to all or most Checkmk installations. In any case there may be additional changes that are relevant to you, for example in checks that you use in your installation.

What will you need to do? Following every update — including patch versions — Checkmk will show you the number and content of incompatible changes and ask you to check and take note of them. Before performing the update is the right time to check for any changes from version 2.3.0 that may have an impact on the update. Since Checkmk 2.3.0, such Werks are no longer carried over, but are deleted during the update after approving a confirmation dialog (Werk #15292).

It is also a good idea to get an overview of the incompatible changes in version 2.4.0 before the update: Open the list of Werks on the Checkmk website. In the description of a Werk, you will find information on what you may need to do to make the change compatible.

After you have carried out the update, the number and content of the incompatible changes will be displayed in the Checkmk interface and you will be asked to check and take note of them. Links help you to check so that you can find out with just a few clicks whether you are using an affected component. If it is certain that a Werk has no impact or you have made the required changes, confirm the Werk with Acknowledge.

3. Update

3.1. Best practices when updating

In this section we describe the best practices that we follow even when updating large Checkmk environments. These are certainly not mandatory in every environment, but they can make the process of updating easier for you.

Updating the Checkmk version

As a prerequisite, before updating to version 2.4.0, version 2.3.0 must have been installed on the Checkmk site.

The update to 2.4.0 currently requires at least 2.3.0p29. However, in the future, a specific 2.4.0 patch version may require a higher 2.3.0 patch version for the update. In general, we recommend updating Checkmk to the latest 2.3.0 patch version first, and only then updating to 2.4.0.

Perform a test run of the update

In large environments, where obviously even restoring a current backup of your Checkmk environment could take quite some time, it is recommended to perform a test with a cloned site before updating the production environment. For this purpose, you can, for example, restore the last regular backup of your site under a different name.

root@linux# omd restore newsite /path/to/backup

Alternatively you can also copy your site using omd cp. For this, however, the site must be stopped for a short time:

root@linux# omd stop mysite
root@linux# omd cp mysite newsite

Then, perform the update first on this new cloned site, for example to check the above mentioned local changes in the new environment.

Temporarily disable agent updates

CEE If you are using the automatic agent updates in the commercial editions, you should consider disabling them temporarily before updating Checkmk so as to be able to later switch to the new agents at the hosts in a controlled manner. To do this, first select Setup > Agents > Windows, Linux, Solaris, AIX and on the following page select the menu item Agents > Automatic updates. By clicking on the Icon to edit a list entry. button in front of the Master switch you can disable the agent update completely:

 Disabling agent update using the master switch.

Following a successful update of Checkmk, you can reactivate the agent updates in the same way.

At this point, we recommend that you initially only reactivate the automatic agent updates for individual hosts or host groups. By doing so, the new agent will not be rolled out to all of your servers right away and you can familiarize yourself with the newly delivered data on a few systems. Also, due to the significant increase in the number of check plug-ins supplied, you might find a whole new set of services that you can then set up properly on the test systems of your choice. You may also need to define new thresholds for new services. If you tackle this on a small scale first, you will be able to minimize unnecessary false positives.

On the Automatic agent updates page above, you can simply enter a few hosts or host groups in the corresponding fields and then re-enable the Master switch.

Options when updating agents to activate on specific hosts.
Important

Remember to remove these restrictions on explicit hosts and host groups again once you are satisfied with the results.

Temporarily disable notifications

You should also consider turning off notifications in the site before the update — for similar reasons to those we explained in the previous section on automatic agent updates. This way you avoid your colleagues from the monitoring team receiving unnecessary notifications.

You can turn off notifications centrally with the main Notifications switch in the Master control snap-in.

It may happen that after the update one or another service is CRIT which had not been the case previously. Take care of new problems after the update for example, in the Overview snap-in.

Important

Don’t forget to turn on the notifications again, e.g. when following the update the number of unhandled problems has leveled off to the level seen before the update.

3.2. Updates in distributed monitoring

There are two different procedures for performing an update of all sites included in a distributed monitoring:

  • Stop all sites, perform the update of all sites in a bulk action, and then restart all sites.

  • Under strict conditions, a mixed operation is possible for a certain period of time, in which the remote sites are updated first, following which version equivalency is restored with the update of the central site.

In particular, if you want to update while the system is running, you should read the notes in the general article on Updates and Upgrades.

3.3. Performing the update

Nothing fundamental has changed with the actual update of the software in Checkmk 2.4.0, i.e. you install the new version, perform the update of the Checkmk site, rectify any possible conflicts and check and confirm the incompatible changes.

Perform the update procedure as described in the article on Updates and Upgrades.

4. Follow-ups

4.1. Changes to the user interface

The Checkmk user interface (GUI), which was completely redesigned with version 2.0.0, has not been changed fundamentally in version 2.4.0. The general procedures, which you are already familiar with from versions 2.0.0 to 2.3.0, can also be used as they are in 2.4.0. However, menus, menu items, icons, and other details have been modified to make new features available — and to improve existing ones.

We will present these changes in this User Guide’s articles — and in the Beginner’s Guide you will find a detailed introduction, including the most important elements of the user interface.

4.2. Updating services

As with every major version, Checkmk 2.4.0 introduces a whole new set of check plug-ins. If you do not use the 'discovery check', i.e. the automatic update of the service configuration via the periodic service discovery, you will have to search for services on quite a number of hosts.

If your hosts are organized accordingly (e.g. in folders), you can generally use the Bulk discovery function for this. This function can be found under Setup > Hosts > Hosts and then in the Hosts > Run bulk service discovery menu.

Service names

Each update of Checkmk will involve changing service names to improve the consistency of naming within the monitoring and documentation of Checkmk. Since changing service names means that rules sometimes need to be modified and historical monitoring data can be lost, Checkmk initially leaves the old names in place for updates. For services where the loss of old monitoring data is acceptable and the effort for adapting rules is manageable, you should switch to new service names as soon as possible.

To do this, go to Setup > General > Global settings > Execution of checks and go through the list Use new service names and identify the services where the checkboxes are not yet active and activate them. After applying the changes, the new service names will be active and a few minutes will pass before you see the defined states of the affected services in the monitoring again.

4.3. Vanished services

In some cases, check plug-ins had to be moved out, or check plug-ins had to switch to new APIs.

Does this affect you? The risk of being affected is present if you monitor hardware with very long lifecycle, especially with special agents. During the preparations for the update from 2.3.0 to 2.4.0, for example, we noticed the special agents for many NetApp devices, which we had to switch from the discontinued ZAPI to the more modern REST API (Werk #16767). For licensing reasons, we had to move the agent for Dell PowerConnect to a separate package available in the Exchange (Werk #15359).

What will you need to do? If services disappear after the update, search the Werks for the manufacturer of the monitored products. In most cases, you will have to switch to new APIs or, in rare cases, use packages from the Exchange to monitor a specific hardware or software.

4.4. Installing Python modules

Does this affect you? This will only apply to you if you have explicitly installed Python modules for special agents or agent-based check plug-ins that you have written yourself or obtained from the Exchange and have removed these in the course of preparing for the update.

What will you need to do? First find out whether the previously uninstalled modules have already been delivered with the new Checkmk version, for example:

OMD[mysite]:~$ pip3 list | grep '^cryptography'

If the module has already been found, mark it as not required in your notes. Install the latest version of any modules that are not included:

OMD[mysite]:~$ pip3 install ecdsa

Then test the check plug-ins or special agents that rely on Python modules installed in the site.

5. The outlook

This chapter covers topics not directly related to the current Checkmk version 2.4.0, rather to future versions in development.

5.1. Old HTTP check will be removed

Since 2.3.0 Checkmk already contains a new active check for HTTP connections and certificates. The Check HTTP web service can be found under Setup > Services > HTTP, TCP, Email, …​ in the Networking box. The new check is significantly better performing, more robust and easier to configure than the previous Check HTTP service. In addition, it is now possible to monitor several things at once with a single rule, for example HTTP response code, response time and certificate validity.

In Checkmk 2.4.0 some rarely used options of the old check have been added to the new check to make migration easier. Furthermore, we provide a migration script that maps existing calls and resulting services 1:1. Use this script to convert services from the old to the new active check, because: With Checkmk 2.5.0 no new rules can be configured for the old check. It will be completely removed later.

Once the migration has been completed, you can achieve significant performance benefits if you merge several rules, which consequently result in several executions of the check. In many cases, you can also generate several services from one rule of the new check.

General information on the new check can be found in the two Werks #15514 and #15515. Werk #17725 describes the migration script. A detailed blog article explains the details of the possibilities, limitations and best practices for preparing and following up the migration.

If you come across applications in which the new check cannot replace the old one, you can display the command line used in the same way as procedure described here. You then have the option of calling the old check or a system-wide installed check_http via Setup > Other services > Integrate Nagios plugins.

5.2. Management boards

The term management board stands for separate plug-in cards or extended BIOS functionality (Baseboard Management Controller/BMC, Management Engine/ME, Lights Out Management/LOM) for monitoring and managing the hardware in addition to the installed operating system.

Up to Checkmk 2.4.0, management boards can be configured in two ways: As a host property or as a separate host. As the configurability as host property is very limited, in the future this option will no longer be available.

Support for Redfish compatible management boards has been added and improved during the lifecycle of Checkmk 2.3.0, initially as an optional MKP (Werk #16589). In Checkmk 2.4.0, Redfish monitoring is an integral part of the software (Werk #17404). We explain additional background information and alternatives in the blog article Monitoring management boards.

On this page