Checkmk
to checkmk.com

1. Introduction

In this article you will find the most important topics relevant to an update of your Checkmk from version 2.0.0 to version 2.1.0.

We recommend that you read through the entire article before updating so that you know exactly what to expect: before, during and after the update.

2. Preparations

This chapter gives you an overview of the topics you should take into consideration before you perform the update. Probably not every topic will be relevant for you: For each topic, you can check the corresponding box for your own reference and immediately move on to the next topic.

2.1. Backup

As with any update to production software, you should check that your backups are up to date before performing the update of Checkmk.

Does this apply to you? Yes.

What do you have to do? If you create your backups automatically via Setup > Maintenance > Backups, check there that your backups are up to date and that the most recent backup jobs have completed without errors.

For more information, see the articles on Backups and on Backing up and restoring sites.

2.2. Linux distribution versions

In Checkmk version 2.1.0, some obsolete versions of the Ubuntu distribution are no longer supported. For users of Debian, SLES, and RHEL (as well as binary-compatible derivatives such as CentOS), there are no changes.

Does this affect you? This will affect you if your Checkmk server has one of the following Ubuntu versions installed that are still supported in 2.0.0:

  • Ubuntu 20.10 Groovy Gorilla.

  • Ubuntu 21.04 Hirsute Hippo

Still supported at release of 2.1.0, but not recommended: For Ubuntu 21.10 (Impish Indri), support with security updates was discontinued by Canonical during the beta phase of 2.1.0. Debian 9 (Stretch) will also no longer receive security updates shortly after the release of Checkmk 2.1.0.

New additions are Debian 11 Bullseye and Ubuntu 22.04 Yammy. For both distributions we also build installation packages of 2.0.0 to simplify the recommended update path.

What do you need to do? Before updating Checkmk, first perform a version upgrade of the Linux distribution. Make sure that the target version of the Linux distribution is supported by Checkmk 2.0.0 and 2.1.0.

To find out which Linux distribution versions Checkmk supports, see the installation article and the download page after selecting the Checkmk version and your Linux distribution.

If it is now necessary to perform a version upgrade of your Linux distribution before updating Checkmk, we recommend the following procedure:

  1. First, stop your Checkmk site:

    root@linux# omd stop mysite
  1. Unmount the site’s tmp directory:

    root@linux# umount /opt/omd/sites/mysite/tmp
  1. Move the entire omd directory so that no unwanted changes can occur to this in the next step. A release upgrade usually removes all packages whose dependencies can no longer be satisfied. One of these packages would be Checkmk. To prevent data loss here, this step is necessary and can be performed using mv.

    root@linux# mv /opt/omd /opt/omd.bak
  1. Perform the Linux version upgrade according to the distributor’s instructions.

  2. After successfully upgrading your Linux distribution, you can now return the omd directory that was moved to safety back into its correct place:

    root@linux# mv /opt/omd.bak /opt/omd
  1. Now install the package from Checkmk that matches the new version of the distribution you are using. Important: The Checkmk version number must be exactly the same as the one that was in use previously. For example, if you were using Checkmk 2.0.0p23 before upgrading the distribution, now install the 2.0.0p23 that matches your new Linux distribution version.

    In the case of Ubuntu, the following command would suffice at this point:

    root@linux# apt install /tmp/check-mk-enterprise-2.0.0p23_0.jammy_amd64.deb

    Note: During installation, specify the full file path to the DEB file via apt install.

When reinstalling Checkmk, follow the detailed installation instructions relevant for your distribution.

2.3. The agents for Linux and Windows

With 2.1.0 Checkmk is introducing two new components, the Agent Controller (on the host), and the Agent Receiver (on the server), for the Linux and Windows agents on x86/64, which allow TLS encrypted transmission of the agent output. The Agent Controller will initially operate in a compatibility mode (legacy pull mode) that also allows access from 2.0.0 sites.

Does this affect you? Yes.

What do you need to do? Identify those Windows hosts in the monitoring that use the Windows server operating systems 2008 R2 and client operating systems 7 or higher on x86_64. The same applies for Linux hosts on x86_64 that use systemd version 220 or later for service management. On these hosts, register with the monitoring server following the update to be able to use TLS encrypted transmission. For more detailed information, see the instructions for Linux or Windows.

2.4. Local files

Local files allow you to customize and extend the functionality provided by Checkmk. These files are located in the local part of the site’s directory structure, i.e. in ~/local. Local files can cause problems when updating, as they may not match the new Checkmk version.

Does this affect you? Since it is not possible for Checkmk to intercept and handle local customizations or any third-party extension during an update, you should check your Checkmk site before an update to see if any local files are used on your system, and if so, which ones.

What do you need to do? Get an overview of the local files on your Checkmk site by running the following command as the site user (where the -L option ensures that symbolic links are also followed):

OMD[mysite]:~$ find -L ~/local -type f

In a freshly-installed Checkmk, you should see only a file named README.TXT listed at this time. Anything beyond that should be at the top of your list for troubleshooting, in case there are problems during the update. Special attention should be paid to check plug-ins that still follow the old check API that was valid until version 1.6.0. We will go into this in more detail in the next chapter.

2.5. The Check API

A new Check API was introduced in Checkmk 2.0.0 which restructured the Check plug-ins to unify and simplify their handling. However, since there are a large number of checks that were programmed for the old Check API which was valid until version 1.6.0, migrating from the old to the new Check API may also be an issue for you in version 2.1.0. The large number of existing checks is also the reason why the old Check API will continue to be supported for a transition period.

Does this affect you? The Check API issue may affect you if you have extended the checks shipped with Checkmk with your own, self-written checks and/or if you use plug-ins from other sources and have downloaded these, e.g. from the Checkmk Exchange. The issue will affect you if one, some or even all of these check plug-ins still follow the old check API. The affected files can be found in your site’s local files.

What do you have to do? We try to make the change to the new Check API as easy and convenient as possible for you:

Just as when updating to version 2.0.0, Checkmk tries to automatically migrate all local plug-ins from the directories ~/local/share/check_mk/checks and ~/local/share/check_mk/inventory to the new Check API when updating to version 2.1.0. This 'automigration' happens at runtime; the plug-in files are not changed. For information on how to view the results following the update, see chapter Verify automigration to the Check API.

While chances are good that the automigration of your check plug-ins will work smoothly, the switch to Python 3, also implemented in version 2.0.0, may require some adaptations. Due to the new Check API and the Python 3 code base, there are now two good reasons to revise your self-written check plug-ins. We will provide you with detailed information on what you need to do in the blogpost on migrating check plug-ins.

Last but not least, the new Check API scores with its improved documentation: There is an introductory article on programming your own check plug-ins as well as the always up to date plug-in API reference documentation. Both can be accessed from the Checkmk interface — in the Help menu of the navigation bar under Developer resources.

2.6. Incompatible and obsolete MKPs

Your monitoring system can be extended quite easily and conveniently via the Checkmk extension packages (MKPs). On the one hand, it can be the case that such MKPs are no longer maintained and may then no longer be compatible with new versions of Checkmk. On the other hand, we keep adding new plug-ins and functional extensions to Checkmk, which is why MKPs sometimes simply become obsolete, and their functionality will be provided by Checkmk itself.

Does this affect you? If you have MKPs installed, for this reason a check of these MKPs is strongly recommended — especially before a major update. This will prevent incompatible packages from interfering with the update, or the creation of duplicate or at least very similar services following the update.

What do you need to do? To do this, check your installed MKPs against our Catalog of Check Plug-ins and remove packages which are now provided natively by Checkmk. You can also use this opportunity to remove MKPs that may have only been installed for a test run. A listing can be found via Setup > Maintenance > Extension packages.

On the command line, you can view installed extensions with the following command:

OMD[mysite]:~$ mkp list
my_mkp_from_exchange
my_personal_mkp

Then for each listed MKP you can subsequently also display the files that belong to the package:

OMD[mysite]:~$ mkp list my_personal_mkp
/omd/sites/mysite/local/lib/check_mk/base/plugins/agent_based/check_mk.py
/omd/sites/mysite/local/share/check_mk/web/plugins/metrics/custom_metrics.py

Should you not be able to assign all packages via the check of the installed MKPs and the comparison with our catalog, the test run of the update, described hereafter is recommended to identify incompatibilities in the process and then before the update remove these from your production monitoring.

2.7. Incompatible changes

As in every Checkmk version, there are alterations to the software in the current version 2.1.0 that may have repercussions on your Checkmk installation. A so-called incompatible change requires that you make manual modifications in order to allow existing functions to continue to run as usual and/or to be able to use new functions.

Does this affect you? In general, there will be incompatible changes that will also affect your Checkmk installation, it is however impossible to make a general statement. In this article we have collected those issues that apply to all or most Checkmk installations. In any case there may be additional changes that are relevant to you, for example in checks that you use in your installation.

What do you need to do? Once you have performed the update, the Checkmk interface will show you the number and content of any incompatible changes, and prompt you to review and take note of them. So you will definitely be confronted with this issue — but only after you have performed the update.

It is therefore a good idea to get an overview of the incompatible changes before the update: Open the list of Werks. (The software development of Checkmk is organized in so-called Werks). In the description of a Werk, you will find instructions on what, if anything, needs to be done to make the change compatible.

The frustrating news: A version’s Werks list is very, very long — even if it contains only the incompatible changes. The comforting news: You will be able to see from the number of changes how much effort we put into improving Checkmk for you.

3. Updates

3.1. Best practises when updating

In this section we describe the best practices that we follow even when updating large Checkmk environments. These are certainly not mandatory in every environment, but they can make the process of updating easier for you.

Update the operating system

The operating system on a Checkmk server should in any case always be up to date. However, before updating Checkmk, it certainly does not hurt to make sure of the operating system’s currency beforehand, by using apt (for Debian and Ubuntu), yum (for Red Hat Enterprise Linux (RHEL) based systems) or zypper (for SUSE Linux Enterprise Server).

Update the Checkmk version

Before updating to version 2.1.0, the Checkmk site must have version 2.0.0 installed.

We have previously advised against omitting an intermediate major version when performing a major update, as there are simply too many changes in between that will hinder a smooth update and almost certainly cause problems. With Werk #13320, this recommendation has now been made into a requirement — and a lock introduced that prevents, for example, a direct update from version 1.6.0 to 2.1.0.

An update to version 2.1.0 does not currently require a specific 2.0.0 patch version. However, again, there are good reasons to start the jump to 2.1.0 only from the latest 2.0.0 patch version, since, for example, a 2.0.0 patch may contain fixes that make it easier to update to version 2.1.0.

For this reason, we recommend updating Checkmk to the latest 2.0.0 patch version first, and only then updating to the 2.1.0.

Perform a test run of the update

In large environments, where obviously even restoring a current backup of your Checkmk environment could take quite some time, it is recommended to perform a test with a cloned site before updating the production environment. For this purpose, you can, for example, restore the last regular backup of your site under a different name.

root@linux# omd restore newsite /path/to/backup

Alternatively you can also copy your site using omd cp. For this, however, the site must be stopped for a short time:

root@linux# omd stop mysite
root@linux# omd cp mysite newsite

Then, perform the update first on this new cloned site, for example to check the above mentioned local changes in the new environment.

Temporarily disable agent updates

CEE If you are using the automatic agent updates, you should consider disabling them temporarily before updating Checkmk so as to be able to later switch to the new agents at the hosts in a controlled manner. To do this, first select Setup > Agents > Windows, Linux, Solaris, AIX and on the following page select the menu item Agents > Automatic updates. By clicking on the Icon to edit a list entry. button in front of the Master switch you can disable the agent update completely:

 Disabling agent update using the master switch.

Following a successful update of Checkmk, you can reactivate the agent update in the same way.

At this point, we recommend that you initially only reactivate the automatic agent update for individual hosts or host groups. By doing so, the new agent will not be rolled out to all of your servers right away and you can familiarize yourself with the newly delivered data on a few systems. Also, due to the significant increase in the number of check plug-ins supplied, you might find a whole new set of services that you can then set up properly on the test systems you choose. You may also need new thresholds for new services. If you tackle this on a small scale first, you will be able to minimise unnecessary false positives.

To do this, you can simply enter a few hosts or host groups in the appropriate fields on the above page and then re-enable the Master switch.

Options when updating agents to activate on specific hosts.

Important: Remember to remove these restrictions on explicit hosts and host groups again once you are satisfied with the results.

Temporarily disable notifications

You should also consider turning off notifications in the pre-update site — for similar reasons to the ones we explained in the previous section on automatic agent updates. This way you avoid your colleagues from the monitoring team receiving unnecessary notifications.

You can turn off notifications centrally with the main Notifications switch in the Master control snapin.

It may will happen that after the update one or another service is CRIT which had not been the case previously. Take care of new problems after the update first. You can also view any unhandled problems, e.g. in the Overview snapin.

Important: Don’t forget to turn on the notifications again, e.g. when the number of unhandled problems after the update has leveled off to the level before the update.

3.2. Updates in distributed monitoring

You have a number of procedures for performing the update of the sites involved in a distributed monitoring, i.e. the central site and the remote sites.

Important: Whichever approach you choose: In this scenario, you should also create backups beforehand.

The recommended, safe procedure is update in one go, where you perform the following steps:

  1. Stop all sites .

  2. Perform the update for all sites.

  3. Restart the updated sites.

There is another variant: the Update on the fly. This procedure is required, among other things, when you manage a large number of distributed sites that cannot be updated in one go for technical or organizational reasons.

When updating on the fly, sites with different Checkmk versions must communicate with each other for a transition period — until the update of all sites involved is complete. For this cooperation to work, the versions involved in such a mixed operation must be compatible.

With Werk #13313, version checks have been added to ensure that the sites involved are using compatible versions. To summarise: Sites must have the same Checkmk major version — with one exception: the Checkmk version of a remote site may be exactly one major version higher than that of the central site. The following table shows the possible combinations when updating from 2.0.0 to 2.1.0:

Central siteRemote siteAllowed?

2.0.0

2.0.0

Yes

2.0.0

2.1.0

Yes

2.1.0

2.0.0

No

2.1.0

2.1.0

Yes

It follows from these rules that when updating on the fly, the central site is updated last. Note that the following also applies when updating in a distributed monitoring: major versions must not be omitted, so that, for example, a connection between a 1.6.0 and a 2.1.0 site is not supported.

Note: Since the feature set of different Checkmk versions differs, some features of Checkmk may not work in this mixed mode or may work only in a limited way. In a mixed operation, problems can occur even with compatible versions, and smooth interaction is only ensured if all sites have the same major version. You should therefore keep the period of a mixed operation with different versions as short as possible and update all sites quickly after starting the update.

3.3. Performing the update

Nothing fundamental has changed with the actual update of the software in Checkmk 2.1.0, i.e. you install the new version, perform the update of the Checkmk site, rectify any possible conflicts and check and confirm the incompatible changes.

Perform the update procedure as described in the article on updates and upgrades.

4. Follow-ups

4.1. Changes to the user interface

Checkmk’s user interface (GUI) was completely redesigned with version 2.0.0. In the new version 2.1.0, on the other hand, it will look very familiar to you, since at first glance there have been very few visible changes made compared to 2.0.0. However, menus, menu items, and other details have been modified to make new features available — and to improve existing ones.

In the articles of this User guide we will present these changes, and in the Beginner’s Guide you will find a detailed introduction to the most important elements of the user interface.

Snapins removed from the sidebar

With Werk #13736, some obsolete and little-used snapins for the sidebar have been removed. There are replacements for almost all of these snapins, mostly via menu items in the navigation bar.

4.2. Verify automigration to the Check API

You can check if all local plug-ins in the ~/local/share/check_mk/checks and ~/local/share/check_mk/inventory directories can be automatically migrated to the new Check API.

Does this affect you? This check is useful if there are check plug-ins in your local files that were programmed conforming with the old Check API that was valid until version 1.6.0.

What do you need to do? Run the following command as a a site user:

OMD[mysite]:~$ cmk -R
Failed to auto-migrate legacy plugin to check plugin: mssql_counters
Please refer to Werk 10601 for more information.

The message above shows that a plug-in could not be migrated automatically. All of the plug-ins on the list will need to be migrated manually. There are a number of reasons why automigration cannot be performed, which you can read about in the aforementioned Werk #10601. You can read how to perform manual migration in the blogpost on migrating check plug-ins.

4.3. Updating services

As with every major version, Checkmk 2.1.0 introduces a whole new set of check plug-ins. If you do not use the 'discovery check', i.e. the automatic update of the service configuration via the periodic service discovery, you will have to search for services on quite a number of hosts.

If your hosts are organized accordingly (e.g. in folders), you can generally use the Bulk discovery function for this. This function can be found under Setup > Hosts > Hosts and then in the Hosts > Discover services menu.

Below we list some areas with the most new features.

The Check_MK and Check_MK Agent services

The two services 'Check_MK' and 'Check_MK Agent' in monitoring.

The Check_MK and Check_MK Agent services still have the same names as in version 2.0.0, but these operate in a somewhat different way in 2.1.0:

  • The Check_MK Agent service is now always present and provides all information related to agent deployment, such as the version of the agent, the time of the last update, the number of agent plug-ins and local checks, or the restriction of IP addresses. Previously, this service only took care of agent updates — and was only available when agent updates were set up.

  • Likewise, the Check_MK service is now always present — previously it was only there when passive services were running on the host. The service Check_MK now exclusively takes care of the operational state of the agents. This service reports on, for example, connection errors or missing data, and also shows the execution time.

Cluster services

With Checkmk version 2.1.0, the way some cluster services behave has changed. Affected services assume the UNKNOWN state and require configuration adaptation.

Checkmk identifies four cluster modes:

  • Best (the value for the best node determines the overall state).

  • Worst (analog: value for the worst node)

  • Failover (only one node is allowed to return OK)

  • Native Clustering via API function (freely programmable, e.g.: 80% of nodes must be OK for the cluster to be OK).

Legacy plug-ins using the old Check API previously had an additional implicit clustering capability, where the output of the assigned hosts was simply evaluated in sequence and the last individual state received determined the overall state of the cluster. Since this behavior was not reliable, it was removed in 2.1.0.

In the course of porting to 2.1.0, we removed the native clustering retrofitted to 2.0.0 from some supplied check plug-ins when it was not useful. Instead, we selected one from the best, worst, or failover behaviors as the default. This list shows the affected plug-ins and their new default behavior. Also affected are plug-ins programmed in-house or obtained from the Checkmk Exchange that do not use API integration for native cluster mode.

All of these services will assume the UNKNOWN state after updating to 2.1.0 if the cluster mode once selected is no longer supported. In these cases, use the Aggregation options for clustered services rule to define one of the three cluster modes best, worst, or failover.

Service descriptions

Each update of Checkmk will involve changing service descriptions to improve the consistency of naming within the monitoring and documentation of Checkmk. Since changing service descriptions means that rules sometimes need to be modified and historical monitoring data is lost, Checkmk initially leaves the old descriptions in place for updates. For services where the loss of old monitoring data is acceptable and the effort for adapting rules is manageable, you should switch to new service descriptions as soon as possible.

To do this, go to Setup > General > Global Settings > Execution of checks and go through the list Use new service descriptions and identify the services where the checkboxes are not yet active and activate them. After applying the changes, the new service descriptions will be active and a few minutes will pass before you see the defined states of the affected services in the monitoring again.

4.4. New ports

To perform TLS registration of hosts in the monitoring, they need access to port 443 (respectively 80) for REST API requests and port 8000 (new component Agent Receiver). If necessary, be sure to configure routing and firewall settings to allow these hosts to access these ports on the Checkmk server.

If security policies prevent you from making these ports available, you can perform a registration by proxy directly on the Checkmk server.

4.5. Registering agents for TLS

After updating the agents to 2.1.0, the Check_MK Agent service assumes WARN status for the affected hosts and displays the message TLS is not activated on monitored host. In the two articles on the Linux and Windows agents, we describe the registration of the certificates in use with the cmk-agent-ctl command line tool. This tool also provides many options for automatic registration — for example, via post-installation snippet in Bakery plug-ins.

If unencrypted data transfer is desired for a transition period for the parallel operation of 2.0.0 servers in a 2.1.0 environment, modify the Checkmk Agent installation auditing ruleset to allow the Check_MK Agent service to assume OK status despite the lack of registration.

4.6. The new Kubernetes monitoring

Kubernetes monitoring has been rewritten from the ground up in Checkmk 2.1.0 and now enables a simplified and efficient monitoring. Unfortunately, because the architecture has fundamentally changed in version 2.1.0, it is not possible to carry over or rewrite any previous monitoring data collected from your Kubernetes objects under previous Checkmk versions.

Does this affect you? This affects you if you used the Checkmk special agent to monitor Kubernetes in version 2.0.0. The old special agent and associated check plug-ins will still work in version 2.1.0, but will subsequently be removed from Checkmk. We therefore recommend a complete move to the new, much improved Kubernetes monitoring in version 2.1.0.

What do you need to do? Completely redeploy Kubernetes monitoring as described in the Monitoring Kubernetes article.

4.7. Encrypting the notification spooler

Does this affect you? If you use distributed monitoring with centralized notification, you will benefit from increased security when enabling data transfer encryption in the mknotifyd notification spooler.

What do you need to do? When updating existing Checkmk installations, connections that have already been configured will initially remain usable without encryption. However, in this case Analyze configuration will fault the setting with CRIT. After migrating all involved sites to 2.1.0, change Global settings > Notification spooler configuration > Encryption to Encrypt communication with TLS. New connections automatically use TLS encryption when they are created.

4.8. The REST API replaces the Web API

The Web API (also known as HTTP API) has been discontinued and will be unavailable in the next Checkmk version. Its replacement is ready as the REST API introduced in version 2.0.0.

Does this affect you? This affects you if you have written custom scripts that use the Web API.

The Web API will continue to work in this Checkmk version, and in existing sites updated from version 2.0.0 to 2.1.0, the Web API will remain enabled. In sites newly created with the 2.1.0 version, the Web API will have already been disabled in the Global settings. The switch to turn the Web API on and off can be found in Setup > General > Global settings > Site management > Disable Web API.

What do you need to do? We recommend migrating all existing scripts that use the Web API to the REST API as soon as possible in order to catch any possible problems early, before upgrading to Checkmk version 2.2.0.

The REST API already provides access to more areas in Checkmk than the Web API, such as host status, service status, scheduled downtimes, time periods, problem acknowledgement, and Business Intelligence (BI).

The functions for accessing sites in a distributed monitoring, and metrics and graphs that are still missing from the REST API will be added in 2.1.0 patch versions, so that over time all functions that could also be used in the Web API will again be available.

5. The outlook

This chapter covers topics not directly related to the current Checkmk version 2.1.0, but to one of its subsequent versions.

5.1. The Web API is being removed

The Web API (also called HTTP API) will be removed in the next Checkmk version 2.2.0. The Web API will be replaced by the REST API introduced in 2.0.0.

5.2. The old Kubernetes special agent will be removed.

The Checkmk 2.0.0 special agent for Kubernetes monitoring and its associated check plug-ins will be removed in the next Checkmk version 2.2.0. Details can be found under the Werk #13561. Replacements are provided in version 2.1.0 by the completely rewritten new Kubernetes monitoring.

5.3. NSCA will no longer be supported

Support for the Nagios Service Check Acceptor (NSCA) will be removed in the next Checkmk version 2.2.0.

NSCA is a Nagios add-on for getting results from passive checks on remote hosts in a distributed Nagios configuration. This deprecated method for setting up distributed monitoring offers no advantages over Checkmk’s own methods.

Details and notes on its replacement can be found in Werk #13644.

5.4. The old HW/SW Inventory plug-in API is no longer supported

Support for the old HW/SW Inventory plug-in API will be removed in the next Checkmk version 2.2.0.

The plug-ins following this API are located in the site directory under ~/share/check_mk/inventory and (as local files) under ~/local/share/check_mk/inventory. These plug-ins will work in version 2.1.0, but will be ignored in version 2.2.0 and later.

If you use custom plug-ins, migrate these to the new Check API before updating to version 2.2.0. See Werk #14084 for more details and information on migration.

On this page