Updating Checkmk is a bit different than most other software packages that you may be familiar with. Why is that?
The reason is that Checkmk not only allows multiple independent instances (sites) to run on a single server, but it also allows multiple software versions to be installed simultaneously. With this system each instance is assigned to an installed version of the software. To illustrate, we can take the following situation on a fictional server:
Here the instance
mysite1 uses version
but the instances
1.6.0p10.cre. The Checkmk-Version
actually installed, but is currently not being used.
This example makes it clear that an update does not simply mean the installation of a new Checkmk RPM/DEB package on a server. Another additional step is also required. Let us take the following situation as an example:
Here the instance
mysite is to be updated to the
1.6.0p13.cee. The first step is to download and install
the appropriate RPM/DEB-Packet. This is performed exactly as with the
initial installation. At first the newly-installed version will
not be used by any instance, and it will look like this:
The second step will now be an update of the instance from
1.6.0p13.cee. This is achieved with the
omd update command,
which we will discuss in detail below:
Following the update, the (likely) no longer required version
can be deleted by uninstalling the appropriate package.
Since we are unable to detect and handle your local customizations or any third-party extensions for Checkmk during an update, before updating your Checkmk instance you should check that these are compatible with your new Checkmk version. Enter the following command to get an overview of the so-called local structure:
find ~/local -follow -type f
In a freshly-installed Checkmk, this command would normally only find a file
README.TXT. Everything else that is displayed should — in the
event of a problem with the update — be available from your
3. Updating of Checkmk in detail
3.1. Installation of newer versions
As described in the introduction, the first step with an update is the installation of the desired new version of Checkmk. This is achieved in exactly the same way as with the initial installation — it will however proceed somewhat more quickly since most of the dependent packages have already been installed. In the following example we are installing the package for Ubuntu 20.04 (Focal Fossa):
root@linux# apt install /tmp/check-mk-enterprise-1.6.0p13_0.focal_amd64.deb
Note: When installing a local package via
apt install, you have to state the full path to the .deb file.
A list of your installed Checkmk-versions can be displayed at any time with the
omd versions command:
root@linux# omd versions 1.5.0p20.cre 1.5.0p24.cee 1.6.0p13.cee (default) 2020.06.10.cee
One of these listed versions is marked with
This default version will be used automatically when creating
new sites, as long as no other version is specified with
omd -V myversion create mysite. When working with existing sites this
is irrelevant. The current default version can be queried with
and it can be altered with
root@linux# omd version 1.5.0p20.cre root@linux# omd setversion 1.6.0p13.cee root@linux# omd version 1.6.0p13.cee
The default version plays no role when updating or managing existing instances.
omd command always starts itself automatically in the version
appropriate to the instance.
A listing of the current instances and the versions they use is provided
omd sites command:
root@linux# omd sites SITE VERSION COMMENTS mysite 1.5.0p24.cre test 1.6.0p13.cee default version
3.2. Performing the update
Once the desired new version has been installed, the instance can be updated.
root-permissions are required for this.
The best way to do this is as an instance user:
root@linux# su - mysite
Ensure that the instance has been stopped:
The update – in effect switching to a different version – can now simply be
performed with the
omd update command:
If more than one target version is available, a selection list will open:
When updating from a Raw Edition to the Checkmk Enterprise Standard Edition you will be alerted to that fact:
An important part of an update is the refreshing of the originally provided configuration files. Here changes that had possibly been made to these files by the user will not simply be discarded, rather they will be merged. This functions very much like version control systems which attempt to amalgamate changes made to a single file simultaneously by multiple developers.
Occasionally — when the changes affect the same location in the file — that won’t function, and a conflict occurs. How you can solve such conflicts will be explained later below.
The update provides a listing of all modified files and directories:
2020-06-16 14:25:20 - Updating site 'mysite' from version 1.6.0p10.cre to 1.6.0p13.cee... * Installed dir var/check_mk/rrd * Installed dir var/check_mk/reports * Installed dir var/check_mk/reports/archive * Installed file etc/logrotate.d/cmc * Installed file etc/logrotate.d/mknotifyd * Installed file etc/logrotate.d/liveproxyd Executing update-pre-hooks script "cmk.update-pre-hooks"...OK Output: Initializing application... Loading GUI plugins... Updating Checkmk configuration... + Rewriting WATO tags... + Rewriting WATO hosts and folders... + Rewriting WATO rulesets... + Rewriting autochecks... + Cleanup version specific caches... Done Finished update.
Once everything has been successfully processed, the instance can be switched to the new version…
omd version 1.6.0p13.cee
… and can then be started:
3.3. Incompatible changes
Software development of course consists of changes. Because we are always actively working to keep Checkmk modern, sometimes cutting dead weight and making changes that turn out to be incompatible is unavoidable. That means that when updating it may possibly be necessary to adapt your configuration, or you should at least check it.
A typical example of such a situation is with new check plug-ins which replace existing plug-ins. If you use one of the affected plug-ins, a fresh service discovery will be required on the affected host.
An overview of all changes in Checkmk, including a search function, can be found online in our Werks. Even more practical however is the built-in search function in the version release notes. This is accessed by clicking on the version number at the top-left of the side bar:
Checkmk tracks new changes automatically, and issues appropriate warnings if they are incompatible:
You can then inspect these ‘Werks’ (‘works’), and approve them with a mouse click. You can also find a listing covering the complete history of the changes, including a search function:
3.4. The update in detail
Are you curious about what exactly is happening ‘under the hood’ of an
update? Or have data conflicts appeared when
omd update is running?
If so, here is some further reading.
Three actions take place during
The refreshing of the default files under
var/– i.e., files created by
The switching of the active version to the target version by changing the symbolic link
versionwhich is found in the Site-directory.
Post-processing by various packages (e.g., Checkmk). In particular, an Activate Changes will be automatically executed in order to generate a valid configuration for the core.
Actualising files, merging changes
The first step is by far the most comprehensive. Here Checkmk demonstrates a big advantage in comparison to the typical software installation — Checkmk helps you to adapt all of the standard configuration files to the prerequisites of the new version. This resembles the procedure for updating a Linux-Distribution, but goes further in the implementation. Checkmk can handle a multiplicity of situations, for example:
The merging of file changes with changes made locally by the user
Files, directories and symbolic links which are obsolete in the new version, or which have been deleted by the user
Changes to permissions
Changes to a file type (a symbolic link derived from a file or directory, or vice versa)
Changes to the target of a symbolic link
Checkmk always ensures that your local changes are retained, and that all of the changes required by the new version are simultaneously implemented.
Merging and conflicts
If the new version intends changing a configuration file on which the user has also made changes, Checkmk automatically attempts to merge both sets of changes. This is achieved using the same methods as used by version-control systems.
The fewest problems are experienced when your and Checkmk’s changes have a clear physical separation in the text (at least a few lines apart). The merge will then be effected automatically, and without needing the user’s intervention.
If two changes ‘collide’ because they both affect the same location in the data, Checkmk cannot and will not decide which of the changes is more important. In such a situation the user will be alerted, and be able to solve the conflict interactively:
In the situation shown above, you now have the following options:
This shows the differences between the new default version and your version of the file in the form of a ‘unified diff’ (
This is similar to the above, but based on the preceeding default version shows which changes you have made to the file.
This third option in effect ‘closes the triangle’ by showing the changes which Checkmk intends making to the file.
By selecting t, your original file – without the already successfully-merged changes – will be opened in an editor. Now edit the file in order to bypass possible conflicts. Once the editor has been closed Checkmk will reattempt the merge.
Here you can decide whether to accept the data ‘as is’. The successfully inserted changes are retained. Apart from this the file remains as customised by the user.
With this you can fall back to the old version of your file, and go without Checkmk’s update for this file. Any customisations that may be required must be performed manually.
Install the new default file: your changes in the old file will be lost.
If you are uncertain, you can open a shell with s. You will find yourself in a directory containing the relevant file, and there can get a picture of the situation. Quit the shell with Strg-D in order to proceed with the update.
Abort the update. The instance retains the old version. Files that have already been changed during the update however remain changed! A new update attempt can be started at any time.
Further conflict situations
Alongside the content-merging of files there is a whole series of further situations in which Checkmk requires your decisions. Some of these are very unusual situations, that nevertheless need to be handled correctly. In these cases Checkmk will always give you the choice of keeping your version, or of adopting the new default version. What is more, there is always the option of aborting an update, or of opening a shell. Examples of such ‘difficult’ situations are:
conflicting changes to file types (e.g., when a file is replaced by a symbolic link)
conflicting changes to file permissions
changed files that are not required by the new sofware version
files, directories or links created by a user, which conflict with a new version’s files/directories/links
Explanation of the tasks in an update
The update procedure will always output a line of explanation when it makes a change to a file automatically. The following situations are possible – files are referred to here, but this also applies analogously to links and directories:
A file has been changed with the new version. Since you have not made a change to the file, Checkmk simply installs the new default version of the file.
A file has been changed with the new version, and at the same time the user has made other changes to the file. Both versions of the file can be merged into one without conflict.
A file has been changed in the new version, and at the same time the user has already made identical changes to the file. Checkmk must not perform any action.
The new version includes a new configuration file which has just now been installed.
The new version includes a file, an identical copy of which the user has already installed.
The new version has obsoleted a file (also applies to a link or a directory). The user has anyway already deleted it. No action.
Another file is obsolete in the new Checkmk, and the user has neither deleted nor changed the existing version. Checkmk deletes this file automatically.
The user has deleted a file which is normally present. Because the version in the new Checkmk has no changes from the last version of the file, Checkmk allows the file to be absent.
The user has already deleted a file, but in the new Checkmk this file contains changes from the previous version. Checkmk installs the new file, and logs a notification of this action to the user.
Checkmk has updated a file’s permissions because different permissions are set in the new version.
3.5. Updating without user interaction
Would you like to automate Checkmk’s software updates? You may at first
have difficulties with the interactive responses from
There is a simple solution for this scenario: the command has options that
have been especially conceived for use in scripts:
omdinhibit all types of “Are you sure… ?” questions.
updatedetermines the desired behaviour if a file conflict occurs.
Possible values for
In the case of a conflict, the user’s own modified version of the file is retained. It is however possible that Checkmk may not be executable, and that manual rectification will be required.
In the event of a conflict, the new standard version of the file will be installed. With this, local changes to the file will be at least partly lost.
In the event of a conflict the update is stopped. That does not necessarily mean that everything will fall back to the old state. A number of configuration files may have already been updated. The software version will however remain the old version.
This is the standard procedure, so in this form the option is actually superfluous.
Below is an example of the complete command for an automated update
1.6.0p13 of the
root@linux# omd stop mysite ; omd -f -V 1.6.0p13.cee update --conflict=install mysite && omd start
omd start a restarting of the
instance will be prevented if the
omd update is aborted by an error.
&& with a semicolon (
if a start should definitely be attempted even in such a situation.
If you are certain that only a single Checkmk-instance is running on the server, the name to be used in a shell script can simply be trapped in a variable:
root@linux# omd sites --bare mysite root@linux# SITENAME=$(omd sites --bare) root@linux# echo $SITENAME mysite
This enables the above line to be independent of the instance’s name. For example, a small shell script could look like this:
#!/bin/bash SITE=$(omd sites --bare) VERSION=1.6.0p13.cee omd stop $SITE omd -f -V $VERSION update --conflict=install $SITE && omd start $SITE
4. Updating the Docker container
Updating a Checkmk docker container from version 1.5.0p13 onward is a breeze. Just follow the appropriate instruction in Checkmk in a Docker container.
For all Checkmk container prior to 1.5.0p13, a little bit more manual labor is required and described in the detailed update instruction for older Checkmk images.
5. Upgrading from the Free Edition to the full version
Was your first installation of Checkmk the Free Edition? Once you have a Standard Edition or Managed Services Edition subscription, you can simply upgrade your existing instances to the full version.
The procedure is exactly the same as that for a ‘normal’ update.
The only difference is that a version’s name with the
is upgraded to a name with the
Simply install the desired package of the full version, and switch the existing
instance to this with
This upgrade can be most easily performed if both versions are identical,
apart from the
.cee suffixes respectively.
What this means for the functionality is that the Free Edition is completely
identical to the full version. Thus an upgrade makes no difference at all.
A simultaneous changeover of the actual version is however quite possible. The fundamentals remain valid as for a normal update of Checkmk.
5.1. Upgrading the Checkmk Appliance
You can also upgrade a demo appliance to a full version with one of the Enterprise Editions without data loss:
6. Upgrading from the Raw Edition to the Enterprise Editions
An upgrade of the Checkmk Raw Edition to the Enterprise Editions is also possible. Here as well, the
procedure is the same as before: install the desired pacakge, and upgrade
the instances with
Since a number of modules and features of the Enterprise Editions are not available in the Raw Edition, following a changeover there are a couple of points to be aware of. The key point is that when creating new instances of Raw Edition or Enterprise Editions different default settings are set.
6.2. Nagios vs. CMC
Since the CRE only supports Nagios as its core, this is preinstalled in instances
created by the CRE. This is retained when an upgrade to the CEE is made.
That means that after an upgrade, processing will initially continue with
a Nagios core. A migration to the CMC is performed with
and this precedure will be described in its own article.
The CEE supports an alternative format for saving historic performance data, one which requires significantly less hard drive-I/O. This is preinstalled in new CEE-instances. CRE-instances will not be changed over automatically by an upgrade. How the migration can be performed is described in its own chapter in the article covering Performance data and graphing.
6.4. Notification spooler
The CRE has no notification spooler. Thus following the changover to the CEE it is not active at first. How to activate it can be learned here.
7. Uninstalling Checkmk
The uninstallation of no longer required Checkmk versions is performed using the operating system’s package manager. To do this, enter the installed package’s name – NOT the file name of the original RPM/DEB file. Important: Only delete Checkmk versions that are no longer being used by any instance!
Checkmk instances that are no longer required can simply be removed with
(thereby deleting all data as well!):
root@linux# omd rm mysite
7.2. SLES, RedHat, CentOS
Here is how to identify which Checkmk packages are being used in RPM-based systems:
root@linux# rpm -qa | grep check-mk check-mk-enterprise-1.5.0p24-el7-38.x86_64 check-mk-enterprise-1.6.0p13-el7-38.x86_64 check-mk-enterprise-2020.06.10-el7-38.x86_64 check-mk-raw-1.5.0p20-el7-38.x86_64
The deletion is performed with
root@linux# rpm -e check-mk-enterprise-1.5.0p24-el7-38.x86_64
7.3. Debian, Ubuntu
Use the below to identify which packets are installed:
root@linux# dpkg -l | grep check-mk ii check-mk-enterprise-1.5.0p24 0.bionic amd64 Check_MK is a full featured system monitoring ii check-mk-enterprise-1.6.0p13 0.bionic amd64 Checkmk - Best-in-class infrastructure & application monitoring ii check-mk-enterprise-2020.06.10 0.bionic amd64 Checkmk - Best-in-class infrastructure & application monitoring ii check-mk-raw-1.5.0p20 0.bionic amd64 Check_MK is a full featured system monitoring
The uninstallation is performed with
root@linux# dpkg --purge check-mk-raw-1.2.8b9 (Read database ... 505719 Files and directories are currently installed.) Remove from check-mk-raw-1.2.8b9 (0.trusty) ... Delete the configuration files from check-mk-raw-1.2.8b9 (0.trusty) ...
8. Fault Diagnosis
If an error occurs when updating Checkmk, it is usually due to one of the following three causes:
You have installed an incompatible extension.
There are incompatible scripts in the local structure of Checkmk.
The manual intervention required by an incompatible item has not been performed.
mkp list mkp find
Check the output of these commands for extensions and files that may have been in your Checkmk site for a long time and which may no longer be needed (e.g. because their function is now provided directly by Checkmk), or which you can no longer assign. To check their influence on the update, you can make copies of these extensions and files and remove them from your check instance — at least as a temporary measure.
9. Files and directories
The files and directories relevant to this article can be found here.
As always, paths that do not begin with
‘/’ apply after the home
directory of the (
Symbolic link to the installation of the Checkmk version used by this instance.
Within this directory a subdirectory exists for every installed Checkmk version. The files belonging to
Within this directory, for every instance there is a home directory containing its configuration files and variable data. This data belongs to the instance’s user, and can be changed by configuration and operations.
Management command for Checkmk instances. This is a symbolic link to the default version’s