Checkmk
to checkmk.com

1. Introduction

Updating Checkmk is a bit different than most other software packages that you may be familiar with. Why is that?

The reason is that Checkmk not only allows multiple independent sites to run on a single server, but it also allows multiple software versions to be installed simultaneously. With this system each site is assigned to an installed version of the software. To illustrate, we can take the following situation on a fictional server:

update1

Here the site mysite1 uses version 2.0.0p3.cee, but the sites mysite2 and mysite3 use version 2.0.0p1.cre. Though Checkmk-Version 2.0.0p1.cfe is installed, it is currently not being used.

This example makes it clear that an update does not simply mean the installation of a new Checkmk RPM/DEB package on a server. Another additional step is also required. Let us take the following situation as an example:

update2

Here the site mysite is to be updated to the Checkmk-version 2.0.0p3.cee. The first step is to download and install the appropriate RPM/DEB-Packet. This is performed exactly as with the initial installation. At first the newly-installed version will not be used by any site, and it will look like this:

update3

The second step will now be an update of the site from 2.0.0p1.cre to 2.0.0p3.cee. This is achieved with the omd update command, which we will discuss in detail below:

update4

Following the update, the (likely) no longer required version 2.0.0p1.cre can be deleted by uninstalling the appropriate package.

2. Before the update

If you are planning to update the Checkmk version 2.0.0 to 2.1.0, you should first read the Update to version 2.1.0 article, in which we have compiled the most important issues you should consider before and after such an update.

But even if you have already installed a 2.1.0 version and want to update to a new stable patch version of 2.1.0, the topics described in the following sections may be relevant.

2.1. Update to major versions

When updating to major versions, you should always update directly to the next version and not just skip any. For example, if you want to update from 1.6.0 to 2.1.0, update to 2.0.0 first. The reason is simple: there are sometimes simply so many changes between two major versions that skipping them can cause problems. This recommendation is enforced by omd update by a block and issues a warning message if necessary. Nevertheless, you can perform such updates by enforcing the operation with the -f option:

OMD[mysite]:~$ omd -f update

Attention: Sites updated in this way are no longer supported by us!

The omd update command also allows updating to a lower version. After such an update in the reverse direction, however, many adjustments are necessary to make configuration and runtime environment compatible again — especially, but not only, when updating to a lower major version. Therefore, we advise against this procedure — and can no longer provide support when updating to a lower version.

2.2. Local files

Local files allow you to customize and extend the functionality provided by Checkmk. These files are located in the local part of the site directory structure, i.e. in ~/local. Local files can cause problems when updating, as they may no longer match the new Checkmk version.

Since it is not possible for Checkmk to intercept and handle local customizations and any third-party extension during an update, you should check your Checkmk site before an update to see if and which local files you are using.

Get an overview of the local files of your Checkmk site by executing the following command as the site user (where the -L option ensures that symbolic links are also followed):

OMD[mysite]:~$ find -L ~/local -type f

In a fresh installation of Checkmk, you will currently only see a file called README.TXT listed. Anything beyond that should be at the top of your troubleshooting list in case you have problems updating.

2.3. Incompatible and obsolete MKPs

The Checkmk extension packages (MKPs) can be used to extend your monitoring system quite easily and conveniently. On the one hand, it happens that such MKPs are no longer maintained and may then no longer be compatible with new versions of Checkmk. On the other hand, we keep adding new plug-ins and functional extensions to Checkmk, which is why MKPs sometimes become obsolete. Their functionality is simply ensured by Checkmk itself.

If you have installed MKPs, it is urgently recommended to check them. This will prevent incompatible packages from interfering with the update, or of having duplicate or at least very similar services following the update.

To do this, check your installed MKPs against our Catalog of Check plug-ins and remove packages which are now provided natively by Checkmk. You can also take this opportunity to remove MKPs that may have only been installed for a test run. You can find a list in the Setup menu under Maintenance > Extension packages. On the command line, with the following commands you can display installed extensions as well as any unpackaged files:

OMD[mysite]:~$ mkp list
OMD[mysite]:~$ mkp find

Should you not be able to assign all packages via the check of the installed MKPs and the comparison with our catalog, we recommend the dry run of the update described below in order to identify incompatibilities and then remove these from your production monitoring before the update.

Check the output of these commands for extensions and files that are no longer needed or that you cannot even allocate. To check their impact on the update, you can make copies of these extensions and files and remove them from your Checkmk site, at least temporarily.

3. Updating Checkmk

3.1. Installing new versions

As described in the introduction, the first step with an update is the installation of the desired new version of Checkmk. This is achieved in exactly the same way as with the initial installation — it will however proceed somewhat more quickly since most of the dependent packages have already been installed. In the following example we are installing the package for Ubuntu 20.04 (Focal Fossa):

root@linux# apt install /tmp/check-mk-enterprise-2.1.0p1_0.focal_amd64.deb

Note: When installing a local package via apt install, you have to state the pull path to the .deb file.

A list of your installed Checkmk versions can be displayed at any time with the omd versions command:

root@linux# omd versions
1.6.0p22.cre
2.0.0p2.cre
2.1.0p1.cee (default)

One of these listed versions is marked with (default). This default version will be used automatically when creating new sites, as long as no other version is specified with omd -V myversion create mysite. The current default version can be queried with omd version, and it can be altered with omd setversion:

root@linux# omd version
OMD - Open Monitoring Distribution Version 2.0.0p3.cee
root@linux# omd setversion 2.1.0p1.cre
root@linux# omd version
OMD - Open Monitoring Distribution Version 2.1.0p1.cre

The default version plays no role when managing existing sites. The omd command always starts with the version appropriate to the site for which the command is called.

A listing of the current sites and the versions they use is provided by the omd sites command:

root@linux# omd sites
SITE             VERSION          COMMENTS
mysite           1.6.0p22.cre
test             2.1.0p1.cee      default version

3.2. Performing the update

Once the desired new version has been installed, the site can be updated. No root-permissions are required for this. The best way to do this is as a site user:

root@linux# omd su mysite

Ensure that the site has been stopped:

OMD[mysite]:~$ omd stop

The update — in effect switching to a different version — can now simply be performed with the omd update command:

OMD[mysite]:~$ omd update

If more than one target version is available, a selection list will open:

update omd update 2

In the following dialogue box, confirm the selected update to the new version:

update omd update 3

When updating from a Raw Edition to the CSE Checkmk Enterprise Standard Edition you will be reminded of this fact:

update raw to enterprise

An important part of an update is the refreshing of the originally provided configuration files. Here changes that had possibly been made to these files by the user will not simply be discarded, rather they will be merged. This functions very much like version control systems which attempt to amalgamate changes made to a single file simultaneously by multiple developers.

Occasionally — when the changes affect the same location in the file — that won’t function, and a conflict occurs. How you can solve such conflicts will be explained later below.

The update provides a listing of all modified files and directories (shortened in the following example):

2022-05-16 16:49:42 - Updating site 'mysite' from version 2.0.0p23.cee to 2.1.0p1.cee...

 * Identical new  etc/omd
 * Installed file etc/mk-livestatus/livestatus.socket
 * Updated        etc/mk-livestatus/nagios.cfg
...
 * Vanished       etc/jmx4perl/config/websphere
 * Vanished       etc/jmx4perl/config
Creating temporary filesystem /omd/sites/mysite_update/tmp...OK
Executing update-pre-hooks script "01_mkp-disable-outdated"...OK
Executing update-pre-hooks script "02_cmk-update-config"...
-| Initializing application...
-| Updating Checkmk configuration...
-| ATTENTION: Some steps may take a long time depending on your installation, e.g. during major upgrades.
-|  1/24 Rewriting password store...
-|  2/24 Migrate Visuals context...
...
-|  24/24 Rewriting InfluxDB connections...
-| Done
OK
Updating core configuration...
Generating configuration for core (type cmc)...
Starting full compilation for all hosts Creating global helper config...OK
 Creating cmc protobuf configuration...OK
Finished update.

Once everything has been successfully processed, the site is switched to the new version:

OMD[mysite]:~$ omd version
OMD - Open Monitoring Distribution Version 2.1.0p1.cee

… and can then be started:

OMD[mysite]:~$ omd start

3.3. Incompatible changes

Software development of course consists of changes. Because we are always actively working to keep Checkmk modern, sometimes cutting dead weight and making changes that turn out to be incompatible is unavoidable. That means that when updating it may possibly be necessary to adapt your configuration, or you should at least check it.

A typical example of such a situation is with new check plug-ins which replace existing plug-ins. If you use one of the affected plug-ins, a fresh service discovery will be required on the affected host.

An overview of all changes in Checkmk, including a search function, can be found online in our Werks.

Even more practical however is the built-in search function in the release notes. After you have logged in to the site, you will find the link highlighted in red in the Help menu:

update help menu incompatible werks

Checkmk keeps track of incompatible changes that occurred during the update from the original to the current version and asks you to check and then acknowledge them:

update incomp werks

You can call up each 'Werk' individually, view it, confirm it with a mouse click — and thus successively reduce the number of open incompatible changes. In addition, you can use the Filter button to show the filter bar and thus have access to the complete history of changes.

3.4. The update in detail

Are you curious about what exactly is happening ‘under the hood’ of an update? Or have data conflicts appeared when omd update is running? If so, here is some further reading.

Three actions take place during omd update:

  1. The refreshing of the default files under etc/ and var/ – i.e., files created by omd create

  2. The switching of the active version to the target version by changing the symbolic link version which is found in the site directory

  3. Post-processing by various packages (e.g., Checkmk). In particular, an Activate Changes will be automatically executed in order to generate a valid configuration for the core.

Updating files, merging changes

The first step is by far the most comprehensive. Here Checkmk demonstrates a big advantage in comparison to the typical software installation — Checkmk helps you to adapt all of the standard configuration files to the prerequisites of the new version. This resembles the procedure for updating a Linux-Distribution, but goes further in the implementation. Checkmk can handle a multiplicity of situations, for example:

  • The merging of file changes with changes made locally by the user

  • Files, directories and symbolic links which are obsolete in the new version, or which have been deleted by the user

  • Changes to permissions

  • Changes to a file type (a symbolic link derived from a file or directory, or vice versa)

  • Changes to the target of a symbolic link

Checkmk always ensures that your local changes are retained, and that all of the changes required by the new version are simultaneously implemented.

Merging and conflicts

If the new version intends changing a configuration file on which the user has also made changes, Checkmk automatically attempts to merge both sets of changes. This is achieved using the same methods as used by version-control systems.

The fewest problems are experienced when your and Checkmk’s changes have a clear physical separation in the text (at least a few lines apart). The merge will then be effected automatically, and without needing the user’s intervention.

If two changes ‘collide’ because they both affect the same location in the data, Checkmk cannot and will not decide which of the changes is more important. In such a situation you are switched on as a user and can interactively resolve the conflict:

omd update

In the situation shown above, you now have the following options:

d

This shows the differences between the new default version and your version of the file in the form of a ‘unified diff’ (diff -u).

y

This is similar to the above, but based on the preceding default version shows which changes you have made to the file.

n

This third option in effect ‘closes the triangle’ by showing the changes which Checkmk intends making to the file.

e

Resolve the conflict manually in an editor.

t

By selecting t, your original file – without the already successfully-merged changes – will be opened in an editor. Now edit the file in order to bypass possible conflicts. Once the editor has been closed Checkmk will reattempt the merge.

k

Here you can decide whether to accept the data ‘as is’. The successfully inserted changes are retained. Apart from this the file remains as customized by the user.

r

With this you can fall back to the old version of your file, and go without Checkmk’s update for this file. Any customizations that may be required must be performed manually.

i

Install the new default file version: your changes in the old file will be lost.

s

If you are uncertain, you can open a shell with s. You will find yourself in a directory containing the relevant file, and there can get a picture of the situation. Quit the shell with Ctrl+D in order to proceed with the update.

a

Abort the update. The site retains the old version. Files that have already been changed during the update however remain changed! A new update attempt can be started at any time.

Further conflict situations

Alongside the content-merging of files there is a whole series of further situations in which Checkmk requires your decisions. Some of these are very unusual situations, that nevertheless need to be handled correctly. In these cases Checkmk will always give you the choice of keeping your version, or of adopting the new default version. What is more, there is always the option of aborting an update, or of opening a shell. Examples of such ‘difficult’ situations are:

  • conflicting changes to file types (e.g., when a file is replaced by a symbolic link)

  • conflicting changes to file permissions

  • changed files that are not required by the new sofware version

  • files, directories or links created by a user, which conflict with a new version’s files/directories/links

Explanation of the output during an update

The update procedure will always output a line of explanation when it makes a change to a file automatically. The following situations are possible – files are referred to here, but this also applies analogously to links and directories:

Updated

A file has been changed with the new version. Since you have not made a change to the file, Checkmk simply installs the new default version of the file.

Merged

A file has been changed with the new version, and at the same time the user has made other changes to the file. Both versions of the file can be merged into one without conflict.

Identical

A file has been changed in the new version, and at the same time the user has already made identical changes to the file. Checkmk must not perform any action.

Installed

The new version includes a new configuration file which has just now been installed.

Identical new

The new version includes a file, an identical copy of which the user has already installed.

Obsolete

The new version has obsoleted a file (also applies to a link or a directory). The user has anyway already deleted it. No action.

Vanished

Another file is obsolete in the new Checkmk, and the user has neither deleted nor changed the existing version. Checkmk deletes this file automatically.

Unwanted

The user has deleted a file which is normally present. Because the version in the new Checkmk has no changes from the last version of the file, Checkmk allows the file to be absent.

Missing

The user has already deleted a file, but in the new Checkmk this file contains changes from the previous version. Checkmk installs the new file, and logs a notification of this action to the user.

Permissions

Checkmk has updated a file’s permissions because different permissions are set in the new version.

3.5. Updating without user interaction

Would you like to automate Checkmk’s software updates? You may at first have difficulties with the interactive responses from omd update. There is a simple solution for this scenario: the command has options that have been especially conceived for use in scripts:

  • The options -f or --force directly following omd inhibit all types of "Are you sure…​ ?" questions.

  • The option --conflict= directly following update determines the desired behavior if a file conflict occurs.

Possible values for --conflict= are:

--conflict=keepold

In the case of a conflict, the user’s own modified version of the file is retained. It is however possible that Checkmk may not be executable, and that manual rectification will be required.

--conflict=install

In the event of a conflict, the new standard version of the file will be installed. With this, local changes to the file will be at least partly lost.

--conflict=abort

In the event of a conflict the update is stopped. That does not necessarily mean that everything will fall back to the old state. A number of configuration files may have already been updated. The software version will however remain the old version.

--conflict=ask

This is the standard procedure, so in this form the option is actually superfluous.

Below is an example of the complete command for an automated update to version 2.1.0p1.cee of mysite:

root@linux# omd stop mysite ; omd -f -V 2.1.0p1.cee update --conflict=install mysite && omd start mysite

Through the && before omd start a restarting of the site will be prevented if the omd update is aborted by an error. Replace the && with a semicolon (;) if a start should definitely be attempted even in such a situation.

If you are certain that only a single Checkmk site is running on the server, the name to be used in a shell script can simply be trapped in a variable:

root@linux# omd sites --bare
mysite
root@linux# SITENAME=$(omd sites --bare)
root@linux# echo $SITENAME
mysite

This enables the above line to be independent of the site’s name. For example, a small shell script could look like this:

update.sh
#!/bin/bash
SITE=$(omd sites --bare)
VERSION=2.1.0p1.cee

omd stop $SITE
omd -f -V $VERSION update --conflict=install $SITE  && omd start $SITE

4. Updating the Docker container

The update of a Checkmk site in the Docker container is very simple. The following are the only requirements:

  • The container is not deleted when the container is stopped — i.e., the --rm option was not used at startup.

  • You know the ID of the volume for the container. Normally you should have given its storage a unique ID when you started the container. If you are unsure of your volume’s ID, you can retrieve information about the container named myContainer with the command docker inspect myContainer.

If you followed the installation guide for Checkmk in Docker you should automatically meet the requirements.

The update process is performed in 3 steps:

  1. Stop the container. If the Checkmk container is called myContainer, the command will be: docker stop myContainer.

  2. Remove the container. The command is: docker rm myContainer.

  3. Start a new container with the command docker container run with the desired version, and mount the known volume. If your volume is called myVolume, the corresponding option is -v myVolume:/omd/sites. All options of the command can be found in installation guide for Checkmk in Docker.

Checkmk will then automatically do the rest — updating and starting your Checkmk site. Afterwards you will be able to log in as usual.

5. Upgrading the Free Edition to the full version

Was your first installation of Checkmk the Free Edition? Once you have a Standard Edition or Managed Services Edition subscription, you can simply upgrade your existing sites to the full version.

The procedure is exactly the same as that for a ‘normal’ update. The only difference is that a version’s name with the .cfe suffix is upgraded to a name with the .cee suffix. Simply install the desired package of the full version, and switch the existing site to this with omd update.

This upgrade can be most easily performed if both versions are identical, apart from the .cfe and .cee suffixes respectively. What this means for the functionality is that the Free Edition is completely identical to the full version. Thus an upgrade makes no difference at all.

A simultaneous upgrade with a change of version is however quite possible. The fundamentals remain valid as for a normal update of Checkmk.

5.1. Upgrading the Checkmk Appliance

You can also upgrade the Free Edition appliance to a full version of one of the Enterprise Editions without data loss. For this you must also upgrade the firmware of the appliance!

  1. First upgrade the firmware of the appliance via its web interface from the demo version to the current firmware of the full version.

  2. Install the Checkmk software of the Standard Edition or the Managed Services Edition on the appliance.

  3. In the site management of the appliance, switch the sites to the full version.

6. Upgrading the Raw Edition to one of the Enterprise Editions

6.1. Introduction

An upgrade of the CRE Checkmk Raw Edition to one of the Enterprise Editions is also possible. Here as well, the procedure is the same as before: install the desired package, and upgrade the sites with omd update.

Since a number of modules and features of the Enterprise Editions are not available in the Raw Edition, following a changeover there are a couple of points to be aware of. The key point is that when creating new sites of Raw Edition or Enterprise Editions different default settings are set.

6.2. Nagios vs. CMC

Since the Raw Edition only supports Nagios as its core, this is preinstalled in sites created by the Raw Edition. This is retained when an upgrade to the CSE Checkmk Enterprise Standard Edition is made. That means that after an upgrade, processing will initially continue with a Nagios core. A migration to the CMC is performed with omd config, and this procedure will be described in its own article.

6.3. RRD-Format

The Enterprise Editions support an alternative format for saving historic performance data, one which requires significantly less hard drive-I/O. This is preinstalled in new Enterprise Editions sites. Raw Edition sites will not be changed over automatically by an upgrade. How the migration can be performed is described in its own chapter in the article covering Performance data and graphing.

6.4. Notification spooler

The Raw Edition has no notification spooler. Thus following the changeover to one of the Enterprise Editions it is not active at first. How to activate it can be learned here.

7. Uninstalling Checkmk

7.1. Overview

The uninstallation of no longer required Checkmk versions is performed using the operating system’s package manager. To do this, enter the installed package’s name – NOT the file name of the original RPM/DEB file. Important: Only delete Checkmk versions that are no longer being used by any site!

Checkmk sites that are no longer required can simply be removed with omd rm (thereby deleting all data as well!):

root@linux# omd rm mysite

7.2. SLES, RedHat, CentOS

Here is how to identify which Checkmk packages are being used in RPM-based systems:

root@linux# rpm -qa | grep check-mk
check-mk-enterprise-2.1.0p1-el8-38.x86_64
check-mk-raw-2.0.0p2-el8-38.x86_64
check-mk-raw-1.6.0p22-el8-38.x86_64

The deletion is performed with rpm -e:

root@linux# rpm -e check-mk-raw-1.6.0p22-el8-38.x86_64

7.3. Debian, Ubuntu

Use the below to identify which packets are installed:

root@linux# dpkg -l | grep check-mk
ii  check-mk-enterprise-2.1.0p1  0.focal  amd64  Checkmk - Best-in-class infrastructure & application monitoring
ii  check-mk-enterprise-2.0.0p3  0.focal  amd64  Checkmk - Best-in-class infrastructure & application monitoring
ii  check-mk-raw-2.0.0p2         0.focal  amd64  Checkmk - Best-in-class infrastructure & application monitoring

The uninstallation is performed with dpkg --purge:

root@linux# dpkg --purge check-mk-raw-1.6.0p22
(Reading database ... 567850 files and directories currently installed.)
Removing check-mk-raw-1.6.0p22 (0.focal) ...
...

8. Fault Diagnosis

If an error occurs when updating Checkmk, it is usually due to one of the following three causes, which have already been mentioned in the previous chapters:

9. Files and directories

The files and directories relevant to this article can be found here. As always, paths that do not begin with ‘/’ apply after the site’s home directory (/omd/sites/mysite).

Path name Function

~/version

Symbolic link to the installation of the Checkmk version used by this site.

/omd/versions

In this directory a subdirectory exists for every installed Checkmk version. The files belonging to root and are never changed.

/omd/sites

In this directory, for every site there is a home directory containing its configuration files and variable data. This data belongs to the site’s user, and can be changed by configuration and operations.

/usr/bin/omd

Management command for Checkmk sites. This is a symbolic link to the default version’s bin-directory. When a particular site is accessed the omd-command substitutes itself with that of the appropriate version.

On this page