Update to version 2.2.0

As with any update to production software, you should check that your backups are up to date before performing the update of Checkmk.

Does this apply to you? Yes.

What do you have to do? If you create your backups automatically via Setup > Maintenance > Backups, check there that your backups are up to date and that the most recent backup jobs have completed without errors.

For more information, see the articles on Backups and on Backing up and restoring sites.

The Checkmk edition family has grown with the brand new Checkmk Cloud Edition. You can read about what this edition has to offer in the Cloud Edition article. This means saying goodbye to the Checkmk Enterprise Free Edition, which is no longer available in 2.2.0.

However, the functionality of the Free Edition lives on in the Cloud Edition: the time-limited trial and the permanent use of one of the commercial editions on a small scale. However, there is a change to the limitation for permanent use: With the Free Edition you can monitor up to 25 hosts in one site, with the Cloud Edition you are limited to monitor 750 services in one site.

Does this affect you? If you are currently using the Free Edition in the 2.1.0 version, you will need to find a new edition when updating to the 2.2.0. A short introduction to the various Checkmk editions can be found here.

What do you need to do? If in your case the 25 host limit in the Free Edition was sufficient, but the 750 free services of the Cloud Edition were not, you can either purchase a subscription and upgrade to one of the commercial editions — or choose to downgrade to the free Raw Edition. Upgrade or downgrade the edition before updating to version 2.2.0. Follow the instructions in the article on Updates and Upgrades, especially for a downgrade to the Raw Edition.

If you don’t have any problems with the new limit of 750 services, simply upgrade your Free Edition to the new Cloud Edition, as described below.

Some obsolete distributions will no longer be supported in the Checkmk 2.2.0 version.

Does this affect you? This will affect you if your Checkmk server has one of the following — still supported in 2.1.0 — Linux distributions installed:

What do you need to do? Before updating Checkmk, first perform a Linux distribution version upgrade. Make sure that the target version of the Linux distribution is supported by Checkmk 2.1.0 and 2.2.0. To find out which Linux distribution versions are supported by Checkmk, see the update matrix for version 2.2.0 and the download page after you have selected the Checkmk version and your Linux distribution.

More recent Ubuntu STS versions will not receive support for the entire product lifecycle of Checkmk 2.2.0. However, we give users of these versions the opportunity to seamlessly update to the next LTS 24.04. Details can be found in the OS support policy.

Checkmk 2.2.0 uses new JavaScript features that are not available in older browsers. Which browsers are supported in which versions can be found in the Release notes.

Does this affect you? You will usually have automatic updates to the latest version enabled on desktop systems.

What do you need to do? Check the browser version you are using and install a more recent browser if necessary. If you are using Single Board Computers, Smart TVs or Digital Signage solutions to display dashboards and you have no control over the system browser, test that any required dashboards are displayed correctly before updating. If necessary, contact the hardware manufacturer for updates.

In Checkmk 2.2.0 the Linux agent no longer supports Python 2.6 for plug-ins.

Does this affect you? Only if you have very old Linux systems in use (Python 2.6 is from October 2008 and was superseded by Python 2.7 as of July 2010) and you use agent plug-ins written in Python on these.

What do you need to do? Identify the affected systems and update the installed agents to the latest available patch version of 2.1.0. If automatic agent updates are configured, disable them before updating the Checkmk site. You can then manually update the individual components — agent script, plug-ins and local checks — after compatibility tests.

In Checkmk 2.2.0, the Windows agent no longer supports Windows 6.0 (Vista and Server 2008 R1). However, on affected systems, the agent from 2.1.0 can still be used.

Does this affect you? Only if you are still using these antiquated Windows systems (support ended April 11, 2017 and January 14, 2020, respectively).

What do you need to do? Identify the affected systems and update the installed agents to the latest available patch version of 2.1.0. If automatic agent updates are configured, disable them before updating the Checkmk site.

Bakery plug-ins include functions in the Agent Bakery’s agent packages. As announced in Werk #14648, support for legacy Bakery plug-ins will be discontinued from version 2.2.0. These are custom plug-ins that were used in version 1.6.0 and earlier, and which have not yet been migrated to the Bakery API introduced in 2.0.0.

Does this affect you? The Agent Bakery and thus the Bakery API are only relevant for users of the commercial editions.

Before the introduction of the Bakery API, the Bakery plug-ins were stored in the site directory under ~/local/share/check_mk/agents/bakery/. To determine whether you have any old plug-ins, you can search for the bakery_info string here.

As of 2.0.0, Checkmk also displays a warning when baking agent packages with legacy Bakery plug-ins. You can also view this afterwards under Setup > Maintenance > Background jobs in the Agent baking section. When baking via the command line with cmk -A the warning is displayed as command output.

What do you need to do? Migrate old Bakery plug-ins to the Bakery API. You can find an introduction to using the API in the Bakery API article.

Checkmk 2.2.0 upgrades Python from 3.10 to 3.11. This leads to a stricter interpretation of regular expressions. Modifiers as 'inline flags' have been marked as deprecated since Python 3.6, and previously resulted in a warning when tested in the Python prompt. In Python 3.11 they are only allowed at the beginning of a regular expression.

Does this affect you? It will most likely affect you if you use the (?i) modifier to indicate that the following regular expression should be evaluated case-insensitively. Since for performance reasons Checkmk in some places merges several regular expressions into a single one, this change may also affect regular expressions (i.e. that begin with modifiers) that when considered alone would also be valid.

What do you need to do? Identify all regular expressions that use inline flags. For this it is usually sufficient to search for the (? string, for example with this command:

Modify any affected regular expressions. Since modifiers are still valid if their scope is specified, you can equivalently rewrite an existing regular expression:

to:

The update of Python from 3.10 to 3.11 also affects any modules installed in a site. In many cases, post-installed modules are incompatible with Python 3.11. In the worst case, obsolete modules overwrite the functionality of the modules supplied by Checkmk.

Does this affect you? This will only affect you when you have explicitly-installed Python modules for special agents or agent-based check plug-ins that you have written yourself or obtained from the Exchange. If you are unsure, carry out the check described in the following section.

What do you need to do? First find out whether Python modules are installed in the site, and if so — identify them. To do this, search for the directories dist-info and egg-info:

Make a note of the installed modules and then uninstall them:

You can find out how to deal with uninstalled Python modules after the update below.

As of version 2.2.0, the use of certain special characters in user names is no longer allowed. With invalid user names an update to 2.2.0 is not possible, because omd update will abort with an error.

User names in Checkmk may only contain letters (as defined by the Unicode database), digits and the following special characters: Hyphen (-), underscore (_), period (.), the 'at' character (@) and dollar sign ($). Only a single letter, digit, underscore or dollar sign is allowed as the first character.

Does this affect you? When creating local users via Setup > Users, Checkmk has been rejecting invalid user names for quite some time, i.e. even before version 1.6.0. So only user names from external connections like LDAP are affected, if at all. As announced in Werk #14393, since version 2.1.0p20 Checkmk has issued warnings for incompatible user names — on the one hand when running omd update and on the other hand in the user administration, which you can call via Setup > Users. The user names are displayed in the ID column and in case of invalid user names there is a warning.

What do you need to do? Update any incompatible user names found at their source.

As of version 2.2.0, the use of insecure and obsolete hash functions for passwords is no longer allowed. This affects the hash functions DES-Crypt, MD5-Crypt, Apr MD5-Crypt and SHA-256 Crypt. Although the latter is still considered secure for passwords, all users' password hashes should be switched to the more modern bcrypt hash function.

A local user whose password has been converted using one of the aforementioned outdated hash functions will no longer be able to log in under version 2.2.0.

Does this affect you? Since version 2.1.0p16 (see Werk #14391) Checkmk has been warning when insecure password hashes are found in the site directory’s ~/etc/htpasswd file. When omd update is executed, a warning message is displayed with all affected (local) user names. Users from external connections like LDAP are not affected.

In 2.1.0, each affected user was then either prompted to choose a new password after logging in, or the hash was automatically updated in the background during login. However, if a user has not logged in at all since then, it is possible that an outdated password hash is still being used. This user will not be able to log in after updating to version 2.2.0.

What do you need to do? To restore access for an affected user, you must reset their password manually. A new password is assigned either in the user administration via Setup > Users, or on the command line with the cmk-passwd <username> command. For password changes, use only the cmk-passwd command and no longer htpasswd, which comes from the Apache installation and which would generate obsolete password hashes without additional options.

On systems that have been updated from Checkmk 1.4.0 to 1.5.0 at some moment in the distant past, notifications may no longer work after updating from 2.1.0 to 2.2.0.

Does this affect you? The problem only occurs on installations that were updated from Checkmk 1.4.0 to 1.5.0 before p8. Furthermore, it only occurs for users who have not been edited and saved in the meantime. For Checkmk installations that were updated from Checkmk 1.4.0 to 1.5.0p8 or later, or were only installed with Checkmk 1.5.0 or later, it does not occur at all.

What do you need to do? Open the user settings of users who should receive messages and save them once - even if supposedly correct settings are displayed. As this is not very practical for a large number of users, you can also make the necessary changes/checks in the configuration file ~/etc/check_mk/conf.d/wato/contacts.mk. Details can be found at work #6417.

Each Checkmk site provides its HTTP services (GUI, APIs) via its own site-specific Apache process. Global access to this site-specific Apache is provided by the System Apache, which opens ports 80 and 443 for external requests, depending on the system configuration.

To learn about the site-specific Apache, System Apache reads a reverse proxy configuration provided by the site user. A malicious site user could now trick System Apache into executing code as the root user, because System Apache is normally started with root privileges first.

To close this security loophole, we need to separate the System Apache configuration from the site user’s access. To remove the privilege escalation, the omd update-apache-config <sitename> command must be run once for each site.

See Werk #14281 for more detailed information on this topic and additional notes, including how to execute the omd update command when it is used to change certain options.

Does this affect you? Since version 2.1.0p7, when running omd update you are prompted to also run omd update-apache-config <sitename>. If you have already done this when updating to 2.1.0, there is nothing further to do when updating to 2.2.0. The previously urgent recommendation now becomes a requirement when updating to the 2.2.0.

What do you need to do? As the root user, run the omd update-apache-config <sitename> command when prompted while running omd update.

As announced in 2.1.0, the Web API (also called HTTP API) has been removed in Checkmk version 2.2.0. The Web API is replaced by the REST API.

Does this affect you? This will affect you if you have written custom scripts that still use the Web API.

What do you need to do? Migrate all existing scripts that use the Web API to the REST API before moving to Checkmk version 2.2.0. See the REST API article for an overview of which REST API endpoints replace the Web API commands.

The Checkmk 2.0.0 special agent and its associated check plug-ins for monitoring Kubernetes are no longer available in 2.2.0. Instead, use the Kubernetes monitoring, rewritten from scratch and greatly improved in the 2.1.0 version.

As announced in version 2.1.0, Checkmk 2.2.0 no longer supports the Nagios Service Check Acceptor (NSCA). NSCA is a Nagios add-on to get results from passive checks on remote hosts in a distributed Nagios configuration. This deprecated method for setting up distributed monitoring offers no advantages over Checkmk’s own methods. Details and notes on its replacement can be found in Werk #13644.

Local files allow you to customize and extend the functionality provided by Checkmk. These files are located in the local part of the site’s directory structure, i.e. in ~/local. Local files can cause problems when updating, if they no longer match the new Checkmk version.

Does this affect you? Since it is not possible for Checkmk to fully ensure compatibility of local customizations when updating, you should check your Checkmk site before an update to see if any local files are used on your system, and if so, which ones.

What do you need to do? Get an overview of unpackaged local files on your Checkmk site by running the following command as the site user:

If files are listed here, you should group files that belong together in extension packages (MKPs). This makes it easier to deactivate en bloc later, should incompatibilities be detected after the update.

In Checkmk 2.2.0 some internally used programming interfaces (APIs) have been rebuilt.

Does this affect you? The issue of APIs may affect you if you have extended the checks shipped with Checkmk with your own, self-written checks and/or if you use plug-ins from other sources. Since the restructured APIs are considered internal and are also not available to plug-ins from the Checkmk Exchange, the number of extensions affected should be small.

What do you need to do? Check third-party extensions for functionality and make any necessary adjustments to import paths.

Affected:

Your monitoring system can be extended quite easily and conveniently via the Checkmk extension packages (MKPs). On the one hand, it can be the case that such MKPs are no longer maintained and may then no longer be compatible with new versions of Checkmk. On the other hand, we keep adding new plug-ins and functional extensions to Checkmk, which is why MKPs sometimes simply become obsolete, and their functionality will be provided by Checkmk itself.

Does this affect you? If you have MKPs installed, for this reason a check of these MKPs is strongly recommended — especially before a major update. This will prevent incompatible packages from interfering with the update, or the creation of duplicate or at least very similar services following the update.

What do you need to do? To do this, check your installed MKPs against our Catalog of Check Plug-ins and remove packages which are now provided natively by Checkmk. You can also use this opportunity to remove MKPs that may have only been installed for a test run. A listing can be found in the commercial editions via Setup > Maintenance > Extension packages.

On the command line, you can view installed extensions with the following command:

Then for each listed MKP you can subsequently also display the files that belong to the package:

Should you not be able to assign all packages via the check of the installed MKPs and the comparison with our catalog, the test run of the update, described hereafter is recommended to identify incompatibilities in the process and then before the update remove these from your production monitoring.

As in every Checkmk version, there are alterations to the software in the current version 2.2.0 that may have repercussions on your Checkmk installation. A so-called incompatible change requires that you make manual modifications in order to allow existing functions to continue to run as usual and/or to be able to use new functions.

Does this affect you? In general, there will be incompatible changes that will also affect your Checkmk installation, it is however impossible to make a general statement. In this article we have collected those issues that apply to all or most Checkmk installations. In any case there may be additional changes that are relevant to you, for example in checks that you use in your installation.

What do you need to do? Once you have performed the update, the Checkmk interface will show you the number and content of any incompatible changes, and prompt you to review and take note of them. So you will definitely be confronted with this issue — but only after you have performed the update.

It is therefore a good idea to get an overview of the incompatible changes before the update: Open the list of Werks. (The software development of Checkmk is organized in so-called Werks). In the description of a Werk, you will find instructions on what, if anything, needs to be done to make the change compatible.

The frustrating news: A version’s Werks list is very, very long — even if it contains only the incompatible changes. The comforting news: You will be able to see from the number of changes how much effort we put into improving Checkmk for you.

In this section we describe the best practices that we follow even when updating large Checkmk environments. These are certainly not mandatory in every environment, but they can make the process of updating easier for you.

There are two different procedures for performing an update of all sites included in a distributed monitoring:

In particular, if you want to update while the system is running, you should read the notes in the general article on Updates and Upgrades.

Nothing fundamental has changed with the actual update of the software in Checkmk 2.2.0, i.e. you install the new version, perform the update of the Checkmk site, rectify any possible conflicts and check and confirm the incompatible changes.

Perform the update procedure as described in the article on Updates and Upgrades.

If you already have the Free Edition in the 2.1.0 version, it is an obvious option to upgrade to the Cloud Edition when you update to the 2.2.0 version, because you can try this Checkmk edition in the 2.2.0 for a limited time or use it permanently on a small scale.

Note: The direct crossgrade described here is only supported for the change from 2.1.0 Free Edition to 2.2.0 Cloud Edition. In all other cases, first update Checkmk and then upgrade the edition. For distributed environments, again pay attention to the order described here.

Entering a license key after 30 days at the latest is only necessary if you want to use the Cloud Edition without restrictions. If you stay below the limit of 750 services in a single site, the Cloud Edition will switch to the license state "Free" after 30 days.

The Checkmk user interface (GUI), which was completely redesigned with version 2.0.0, has not been changed fundamentally in version 2.2.0. The general procedures, which you are already familiar with from versions 2.0.0 and 2.1.0, can also be used as they are in 2.2.0. However, menus, menu items, icons, and other details have been modified to make new features available — and to improve existing ones.

We will present these changes in this User Guide’s articles — and in the Beginner’s Guide you will find a detailed introduction, including the most important elements of the user interface.

As with every major version, Checkmk 2.2.0 introduces a whole new set of check plug-ins. If you do not use the 'discovery check', i.e. the automatic update of the service configuration via the periodic service discovery, you will have to search for services on quite a number of hosts.

If your hosts are organized accordingly (e.g. in folders), you can generally use the Bulk discovery function for this. This function can be found under Setup > Hosts > Hosts and then in the Hosts > Run bulk service discovery menu.

Since Checkmk 2.1.0, hosts need access to the following ports on the Checkmk server to perform TLS registration: port 443 (80 respectively) for REST API requests and port 8000 for the Agent Receiver. If security policies speak against making these ports available, you can perform a registration by proxy directly on the Checkmk server when using pull mode.

To use the Cloud Edition’s push mode introduced in Checkmk 2.2.0, port 8000 of the Checkmk server must be accessible from the monitored host.

Checkmk 2.1.0 used certificates with de facto unlimited validity to establish the TLS connection. In Checkmk 2.2.0, the certificate lifetime is customizable and set to a default value of five years (see Werk #14964). In addition, a custom Certificate Authority can also be used. Certificates created before the update to version 2.2.0 are automatically replaced by newly generated certificates once the maximum runtime has expired.

Therefore, update Checkmk 2.1.0 agents that use the Agent Controller with encryption to the 2.2.0 agent in a timely manner.

In the few cases where updating the agent is not possible — for example, because the operating system version is no longer supported by the 2.2.0 agent — you must manually re-register affected hosts after the certificate expires. The same applies if hosts running the 2.2.0 agent are inactive for a longer period of time.

Does this affect you? This will affect you if you have made use of the feature to keep MKPs in differing versions.

What do you need to do? During the update, incompatible packages will be automatically set to the Disabled state, but not completely deleted. Packages compatible with the new Checkmk version will be automatically enabled, and you will see the Enabled (active on this site) state.

Once the functionality of the new packages has been verified and the old ones are no longer needed for reference, you can search for obsolete packages:

Then uninstall them with mkp remove.

Does this affect you? This will only apply to you if you have explicitly installed Python modules for special agents or agent-based check plug-ins that you have written yourself or obtained from the Exchange and have removed these in the course of preparing for the update.

What do you need to do? First find out whether the previously uninstalled modules have already been delivered with the new Checkmk version, for example:

If the module has already been found, mark it as not required in your notes. Install the latest version of any modules that are not included:

Then test the check plug-ins or special agents that rely on Python modules installed in the site.

mod_auth_mellon is a software module for Apache that provides services for authentication via the Secure Assertion Markup Language (SAML). Up until Checkmk version 2.1.0, logging into Checkmk via SAML was only possible with the help of mod_auth_mellon. As of version 2.2.0, SAML authentication is built into the Checkmk commercial editions and the configuration is done through the Checkmk user interface, as described in the article on SAML. SAML authentication via mod_auth_mellon is also still supported in version 2.2.0.

However, this will change starting with version 2.3.0 — mod_auth_mellon will then no longer be shipped with the Checkmk software. If you would like to continue using this approach to SAML authentication from version 2.3.0 onwards, you will need to install mod_auth_mellon manually. That said, from version 2.3.0, we will no longer provide support for this method.

In version 2.3.0 it will no longer be possible to override the default values in check plug-ins or other global variables using mk files.

This deprecated method for configuring plug-ins has long been undocumented — and in any case no longer works for all of those plug-ins that have been switched to the Check API that was introduced with version 2.0.0. Therefore, from version 2.3.0 configuring check plug-in defaults with mk files is no longer supported.

For details and notes on the changeover, see Werk #15250.

The Prometheus special agent provides kube-state-metrics as a data source, but its checks are no longer actively supported. These have since been replaced by improved counterparts in the Kubernetes agent (see Werk #14572). In addition, in the Prometheus and Alertmanager rules, the specification of IP address/host name, port and file path prefix is replaced by a single input field Custom URL (see Werk #14573).

In both cases, the old method still works in version 2.2.0. However, to use it in version 2.3.0, the configuration must be changed to the new procedure.