Updates and Upgrades

When updating to a higher major version, you must always update step-by-step through any available intermediate versions until the target version has been reached, and not just skip over any intermediate versions. For example, if you want to update from version 2.1.0 to version 2.3.0, first update to the intermediate version 2.2.0. The reason for this procedure is simple: there are sometimes simply so many changes between two major versions that skipping versions can cause problems.

The omd update command also allows an 'update' to a lower version. This procedure is only intended for regressions. After such an update in the reverse direction, many adjustments will be required to make the configuration and runtime environment compatible again — especially, but not only, in the case of an 'update' to a lower major version. We therefore strongly advise against such a procedure — and will also no longer provide support in the case of an update to a lower version.

Your monitoring system can be extended quite easily and conveniently using the Checkmk extension packages (MKPs). On the one hand, it is possible that some older MKPs are no longer being maintained and thus may no longer be compatible with newer versions of Checkmk. On the other hand, we keep adding new plug-ins and functional extensions to Checkmk, which is why MKPs sometimes become obsolete. The functionality of those plug-ins and extensions is simply provided by Checkmk itself as standard.

For this reason, when preparing for an update it is strongly recommended that you review all installed MKPs. This will prevent incompatible packages from interfering with the update, or resulting in duplicate or at least very similar services following the update.

To do this, check your installed MKPs against our Catalog of Check Plug-ins and remove any packages containing functions that are now provided natively by Checkmk. You can also use this opportunity to remove MKPs that may have only been installed for a test run. A list can be found in the Setup menu under Maintenance > Extension packages. On the command line, you can display the installed extensions with the mkp list command. Check the output of this command for extensions that are no longer needed or that you cannot even identify.

Checkmk supports the installation of MKPs for a version newer than the currently running version in preparation for future updates. When performing an update, the package for the lower Checkmk version is then disabled and the package for the higher version is enabled. Details are explained in the article on using MKPs.

Caution: If you have made local changes to files that originally came from MKPs, repack the MKP after increasing the version number. During an update, files that have been otherwise modified will be overwritten by those contained in the MKP.

Local files allow you to customize and extend the functionality provided by Checkmk. These files are located in the local part of the site directory structure, i.e. in ~/local. Local files can cause problems when updating, as they may no longer match the new Checkmk version.

Since it is not possible for Checkmk to intercept and handle local customizations and any third-party extension during an update, you should check your Checkmk site before an update to see if and which local files you are using.

Get an overview of the local files of your Checkmk site by executing the following command as the site user (where the -L option ensures that symbolic links are also followed):

In a fresh installation of Checkmk, you will currently only see a file called README.TXT listed. Anything beyond that should be at the top of your troubleshooting list in case you have problems updating.

Ideally, local files are already fully packaged in MKPs. Use mkp find to identify unpackaged files. For further details on creating packages, see our article on Checkmk extension packages. Once packaged, each extension can be deactivated or reactivated as a complete element.

We don’t need to remind you of the importance of creating a backup immediately before any update, so that you don’t risk losing too much of your monitoring history in the event of a failure during the update. What is relevant at this point is that a regular backup can also serve you well for test runs of a pending update. This practice enables you to restore the backup under an alternative name — and then use the newsite site to test the update before it goes live:

Alternatively you can also copy your site via omd cp. For this, however, a live site must be stopped for a short time:

Then run the update on this new, cloned site first, for example, to check that the local changes mentioned above have been made in the new environment. If the tests with the cloned site were successful, you will usually want to delete or at least stop it before the actual update of the production site for space and performance reasons.

As described in the introduction, the first step with an update is the installation of the desired new version of Checkmk. This is achieved in exactly the same way as with the initial installation — it will however proceed somewhat more quickly since most of the dependent packages have already been installed. In the following example we are installing the package for Ubuntu 22.04 (Jammy Jellyfish):

Note: When installing a local package via apt install, you have to state the pull path to the .deb file.

A list of your installed Checkmk versions can be displayed at any time with the omd versions command:

One of these listed versions is marked with (default). This default version will be used automatically when creating new sites, as long as no other version is specified with omd -V myversion create mysite. The current default version can be queried with omd version, and it can be altered with omd setversion:

The default version plays no role when managing existing sites. The omd command always starts with the version appropriate to the site for which the command is called.

A listing of the current sites and the versions they use is provided by the omd sites command:

Once the desired new version has been installed, the site can be updated. No root-permissions are required for this. The best way to do this is as a site user:

Ensure that the site has been stopped:

The update — in effect switching to a different version — can now simply be performed with the omd update command:

If more than one target version is available, a selection list will open:

In the following dialog box, confirm the selected update to the new version:

To be on the safe side, when you perform an edition upgrade from an Checkmk Raw to one of the commercial editions at the same time as the version update, you will be reminded of this fact again:

An important part of an update is the refreshing of the originally provided configuration files. Here changes that had possibly been made to these files by the user will not simply be discarded, instead they will be merged. This functions very much like version control systems which attempt to amalgamate changes made to a single file simultaneously by multiple developers.

Occasionally — when the changes affect the same location in the file — that won’t function, and a conflict occurs. How you can solve such conflicts will be explained later below.

The update provides a listing of all modified files and directories (shortened in the following example):

If the Completed verifying site configuration line highlighted in the above output is displayed, the site has been switched to the new version:

… and can then be started:

Software development of course consists of changes. Because we are always actively working to keep Checkmk modern, sometimes cutting dead weight and making changes that turn out to be incompatible is unavoidable. That means that when updating it may possibly be necessary to adapt your configuration, or you should at least check it.

A typical example of such a situation is with new check plug-ins which replace existing plug-ins. If you use one of the affected plug-ins, a fresh service discovery will be required on the affected host.

An overview of all changes in Checkmk, including a search function, can be found online in our Werks.

Even more practical however is the search function built into Checkmk. After you have logged in to the site, you will find a link highlighted in red at the top of the Help menu with the number of incompatible and not yet confirmed changes:

Checkmk keeps track of incompatible changes that occurred during the update to the current version and asks you to review and then acknowledge these:

If you open this page via the red link in the Help menu, you will only see the 'Werk' (i.e. the changes) for which something needs to be done and which are therefore marked with Incompatible - TODO. You can call up each Werk individually, view it, confirm it with a mouse click — and thus successively reduce the number of open incompatible changes. In addition, the Help > Change log (works) menu item gives you access to the complete history of changes in the current major version.

Are you curious about what exactly is happening ‘under the hood’ of an update? Or have data conflicts appeared when omd update is running? If so, here is some further reading.

Three actions take place during omd update:

Would you like to automate Checkmk’s software updates? You may at first have difficulties with the interactive responses from omd update. There is a simple solution for this scenario: the command has options that have been especially conceived for use in scripts:

Possible values for --conflict= are:

Below is an example of the complete command for an automated update to version 2.3.0p1.cee of mysite:

Through the && before omd start a restarting of the site will be prevented if the omd update is aborted by an error. Replace the && with a semicolon (;) if a start should definitely be attempted even in such a situation.

If you are certain that only a single Checkmk site is running on the server, the name to be used in a shell script can simply be trapped in a variable:

This enables the above line to be independent of the site’s name. For example, a small shell script could look like this:

The technical reason for the updating approach as described above lies in the protocols used: The central site accesses the data on the remote sites primarily by reading via Livestatus, and in the case of a central configuration, with additional write access via a non-public HTTP API. In both situations, it is the case that new versions introduce real supersets of the protocols used. Thus, an older central site uses only a true subset of the functionality of the newer remote sites. If the central site were updated first, it might issue API calls or Livestatus requests to the remote sites that these do not yet 'understand'.

The maximum version difference of one major version again results from the fact that the removal of interfaces is accompanied by a grace period of exactly one version. For example, the Web API was already no longer used internally by Checkmk 2.1.0, but its removal did not occur until version 2.2.0. For this reason, a 2.1.0 central site works with 2.2.0 remote sites, but a 2.0.0 central site will not work with 2.2.0 remote sites.

To facilitate such phased updates, Checkmk provides the ability to store extension packages with the same name in different versions — one matching the older central site, one matching the newer remote sites — for example. The appropriate version will be activated for each site. Details are described in the article on extension packages (MKPs).

With the viewer sites extension, the viewer sites may be updated only after the sites whose data they display. If a viewer site only shows data from remote sites, it may be updated as soon as these are updated. If, on the other hand, it also shows data from the central site, the viewer’s update may only take place last.

This chapter primarily covers an upgrade to Checkmk Enterprise. You can also upgrade to Checkmk Cloud in one step, but in this case, also refer to the notes in the following section.

Since the commercial editions have quite a few additional modules and features, there are a few things to keep in mind following any upgrade. The crucial point is that when creating new sites in Checkmk Raw or one of the commercial editions, different default settings are set.

When it comes to the monitoring core and the notification system, there are no differences between Checkmk Enterprise and Checkmk Cloud. Depending on the focus of the deployment, you will often use the larger feature set only when adding new hosts. In some places, however, it is still advisable to review the existing settings.

For a complete overview of the additional functionality, see the article on Checkmk Cloud.

Note that in distributed environments, the version update must always be performed first before the edition upgrade can follow. A different sequence or a crossgrade (update of the version and upgrade of the edition in a single action) is not supported. As a general procedure, we recommend the offline upgrade, where you can jump from any lower edition to any higher edition.

For two frequently requested upgrade scenarios (Checkmk Raw to Checkmk Enterprise and Checkmk Enterprise to Checkmk Cloud), mixed operation is possible for a limited period of time. While the offline upgrade works with all versions, the two online scenarios require at least Checkmk 2.2.0p23.

In preparation for the downgrade from Checkmk Cloud to Checkmk Enterprise, you must make at least the following changes:

Furthermore, some cloud services and dashboards will no longer be available. As a result, you will need to clean up vanished services.

If under Checkmk Cloud you used the Grafana plug-in from the 'Grafana Store', you will need to replace it with one installed from the zip archive.

An overview of the differences between Checkmk Cloud and Checkmk Enterprise is provided by the Checkmk Cloud article.

In preparation for downgrading from Checkmk Enterprise to Checkmk Raw, you will need to make at least the following changes:

In addition, some dashboards, graph settings, notification plug-ins and special agents are no longer available. With the Checkmk Enterprise article you can determine how much Checkmk Enterprise functionality will be lost in the event of a downgrade to Checkmk Raw and where you may need to make further adjustments.

The downgrade scenarios presented in the two previous sections can also be carried out in distributed environments. Note that in distributed environments, the version update must always be performed first before the edition downgrade can follow. A different sequence or a crossgrade (update of the version and downgrade of the edition in a single action) is not supported.

There is no downgrade scenario in which Checkmk supports a mixed operation between differing editions. For this reason, we strongly recommend that you perform the edition downgrade offline. To do this, proceed as follows:

Of course, you can wait until all sites have been downgraded before restarting them, which will somewhat reduce the number of notifications generated.

The uninstallation of no longer required Checkmk versions is performed using the operating system’s package manager. To do this, enter the installed package’s name – not the file name of the original RPM/DEB file.
Important: Only delete Checkmk versions that are no longer being used by any site!

Checkmk sites that are no longer required can simply be removed with omd rm (thereby deleting all data as well!):

Here is how to identify which Checkmk packages are being used in RPM-based systems:

The deletion is performed with rpm -e:

Use the below to identify which packets are installed:

The uninstallation is performed with dpkg --purge: