Checkmk
to checkmk.com

1. Introduction

In this article you will find the most important topics relevant to updating your Checkmk version 1.6.0 to 2.0.0.

We recommend that you read through the entire article before the update so that you know exactly what to expect — before, during and after the update.

2. Preparations

This chapter gives you the overview of the topics you should take care of before you perform the update. Not every topic will probably be relevant for you: You can tick one internally and move on to the next topic.

2.1. Backup

As before updating any production software, you should check that your backups are up to date before updating Checkmk.

Does this affect you? Yes.

What do you need to do? If you create your backups automatically via the WATO > Backups module, check there whether the last backup jobs have run through without errors.

For more information, see the articles on Backups and on Backing-up and restoring sites.

2.2. DokuWiki

The DokuWiki is no longer part of the Checkmk software.

Does this affect you? This affects you if you have previously used the wiki, for example, to provide information in the form of notes or via the sidebar snapin WIKI in Checkmk.

What do you need to do?

An important note beforehand: The DokuWiki data stored in Checkmk will not be deleted when updating to Checkmk 2.0.0. Only the software itself will be removed from the corresponding directories. You can therefore still move your DokuWiki data to version 2.0.0 at any time after the update.

If you have previously used the DokuWiki supplied and wish to continue using the information, we recommend installing DokuWiki separately. You will find instructions on how to do this on the dokuwiki.org project pages.

Since all articles and other texts and structures in DokuWiki consist only of text files and directories, moving them to another server is actually extremely simple. To do this, simply copy the contents of the ~/var/dokuwiki/data folder from your Checkmk site to the directory of the same name in your new DokuWiki installation.

If you have been accessing the information in DokuWiki primarily via the Notes icon notes icon in your Service and Host views, you should also edit the URL in the Notes URL for Hosts and Notes URL for Services rules after moving DokuWiki. If you have created a Custom Icon for this purpose instead, change the URL accordingly in Global Settings > Custom icons and actions.

2.3. Linux distribution versions

In Checkmk version 2.0.0, some obsolete Linux distribution versions will no longer be supported.

Does this affect you? This affects you if your Checkmk server has any of the following versions installed that are still supported in 1.6.0:

  • Red Hat Enterprise Linux (RHEL) and CentOS version 6.x

  • SUSE Linux Enterprise Server (SLES) version 12 SP1 or SP2

  • Debian version 8.0

  • Ubuntu version 14.04, 18.10, 19.04 or 19.10

What do you need to do? Before updating Checkmk, first update the Linux distribution. Make sure that the target version of the Linux distribution is supported by Checkmk 1.6.0 and that 2.0.0.

You can find out which Linux distribution versions Checkmk supports in the installation article and on the download page after you have selected the Checkmk version and your Linux distribution.

If it is now necessary to upgrade Linux before updating Checkmk, we recommend the following procedure:

  1. First stop your Checkmk site:

    root@linux# omd stop mysite
  1. Unmount the site’s tmp directory:

    root@linux# umount /opt/omd/sites/mysite/tmp
  1. Move the entire omd directory so that no unwanted changes can occur in the next step. During a release upgrade, all packages whose dependencies can no longer be fulfilled are usually removed. Checkmk would also belong to these packages. To prevent any possible data loss here, this step is necessary and can be carried out via mv.

    root@linux# mv /opt/omd /opt/omd.bak
  1. Upgrade Linux according to the instructions from the respective distributor.

  2. After the successful upgrade of your Linux distribution, you must now move the omd directory that you temporarily put into a safe place back into its correct location:

    root@linux# mv /opt/omd.bak /opt/omd
  1. Now install the package from Checkmk that matches the new version of the distribution you are using. Important: The version number of Checkmk must be exactly the same as the one used before. So if you were using Checkmk 1.6.0p22 before upgrading the distribution, now install the 1.6.0p22 that matches your new Linux distribution version.

    In the case of Ubuntu, the following command would suffice at this point:

    root@linux# apt install /tmp/check-mk-enterprise-1.6.0p22_0.focal_amd64.deb

    Note: During installation, specify the full path to the .deb file via apt install.

When reinstalling Checkmk, follow the detailed installation instructions for your respective distribution.

2.4. Software for forecast graphs

In order to use the feature to forecast graphs in Checkmk 1.6.0, the installation of additional software (among others Prophet from Facebook) is necessary. As of 2.0.0, Checkmk has a completely stand-alone implementation of this prediction function, without the need to install additional software packages.

Does this affect you? If you have been using the forecast graphs introduced in Checkmk 1.6.0, it is highly recommended that you remove the Python and Prophet installed for it from your sites.

What do you need to do? The script you used to install Prophet and its dependencies will also remove these packages for you. To do this, simply call it with the -u option. In distributed monitoring, you will need to repeat this step on each site where you have installed Prophet.

OMD[mysite]:~$ bin/install_fbprophet -u

2.5. Local files

Local files allow you to customize and extend the functionality provided by Checkmk. These files are located in the local part of the site directory structure, i.e. in ~/local. Local files can cause problems when updating, as they may no longer match the new Checkmk version.

Does this affect you? Since it is not possible for Checkmk to intercept and handle local customizations and any third-party extension during an update, you should check your Checkmk site before an update to see if and which local files you are using.

What do you need to do? Get an overview of the local files of your Checkmk site by executing the following command as the site user (where the -L option ensures that symbolic links are also followed):

OMD[mysite]:~$ find -L ~/local -type f

In a fresh installation of Checkmk, you will currently only see a file called README.TXT listed. Anything beyond that should be at the top of your troubleshooting list in case you have problems updating. Scripts and check plug-ins written in Python 2 and check plug-ins that follow the old Check API deserve special attention. We will go into both topics in more detail in the following two chapters.

2.6. Code base Python 3

Most of Checkmk is written in the Python programming language — for over 10 years in Python 2. With version 2.0.0 we have now taken the big step and ported Checkmk to Python 3. This step was costly, but necessary to be able to develop Checkmk efficiently and safely.

Up to and including version 1.6.0, Checkmk comes with Python 2.7 and all necessary third-party modules. Instead of Python 2.7, Checkmk 2.0.0 will be shipped with Python 3.8 and all necessary third party modules.

Does this affect you? This Python version change is relevant for you if you use local files (scripts or plug-ins) written in Python 2.

What do you need to do? In scripts, change the 'Shebang' and adapt the code from Python 2 to 3:

In a script that is called directly from a cronjob or the core, e.g. a notification script, you have often used the following shebang in the past:

mylocalscript
#!/usr/bin/env python

This means that you have used the Python 2.7 delivered in version 1.6.0. Since this no longer exists, the Python version of the operating system is used without changing this line. Which one this is depends on your operating system and can be queried by you with the following command.

OMD[mysite]:~$ python --version
Python 2.7.18

We recommend changing the shebang to:

mylocalscript
#!/usr/bin/env python3

This will allow you to go back to using the Python that comes with Checkmk 2.0.0.

After that, you may need to make some adjustments in your scripts to change your code from Python 2 to Python 3. You will learn how to do this in the following chapter on the new Check API.

2.7. The new Check API

Checkmk 2.0.0 introduces a new Check API that restructures the check plug-ins to standardize and simplify their handling.

However, as there are a large number of checks that were programmed according to the old check API valid up until version 1.6.0, the old Check API will continue to be supported for a transitional period.

Does this affect you? The new API affects you if you have extended the checks delivered with Checkmk with your own, self-written checks and/or if you use plug-ins from other sources and have downloaded them, for instance, from the Checkmk Exchange. The affected files can be found in your site’s local files.

What do you have to do? We have tried to make the change to the new Check API as easy and comfortable as possible for you:

After updating to version 2.0.0 Checkmk tries to migrate all local plug-ins from the ~/local/share/check_mk/checks and ~/local/share/check_mk/inventory directories automatically to the new Check-API. This 'automigration' happens at runtime; the plug-in files are not changed. You can find out how to view the results after the update in the chapter Check automigration to the new Check API.

Although the chances are good that the automigration of your check plug-ins will work smoothly, the changeover to Python 3 may require some adjustments. Due to the introduction of the new Check API and the change of the code base to Python 3, there are two good reasons to revise your self-written check plug-ins. We provide detailed information on the necessary steps for this in the Blogpost on the migration of check plug-ins.

Last but not least, the new check API scores with improved documentation. There is an introductory article on Programming your own check plug-ins with the new check API, and you can access the always up-to-date plug-in API reference documentation in Checkmk 2.0.0 via the Checkmk interface — in the new Help menu in the navigation bar.

2.8. Incompatible and obsolete MKPs

The Checkmk extension packages (MKPs) can be used to extend your monitoring system quite easily and conveniently. On the one hand, it happens that such MKPs are no longer maintained and may then no longer be compatible with new versions of Checkmk. On the other hand, we keep adding new plug-ins and functional extensions to Checkmk, which is why MKPs sometimes become obsolete. Their functionality is simply ensured by Checkmk itself.

Does this affect you? If you have installed MKPs, it is urgently recommended to check them, especially before such a large update. This will prevent incompatible packages from interfering with the update, or of having duplicate or at least very similar services following the update.

What do you need to do? To do this, check your installed MKPs against our Catalog of Check plug-ins and remove packages which are now provided natively by Checkmk. You can also take this opportunity to remove MKPs that may have only been installed for a test run. You can find a list via the WATO module Extension Packages. On the command line, with the following commands you can display installed extensions as well as any unpackaged files:

OMD[mysite]:~$ mkp list
OMD[mysite]:~$ mkp find

Should you not be able to assign all packages via the check of the installed MKPs and the comparison with our catalog we recommend the dry run of the update described below in order to identify incompatibilities and then remove these from your production monitoring before the update.

2.9. Incompatible changes

As in every Checkmk version, there are changes to the software in the current version 2.0.0 that may have repercussions on your Checkmk installation. A so-called incompatible change may require you to make manual adjustments so that existing functions continue to run as usual and/or new functions can be used.

Does this affect you? In general, there will be incompatible changes that will also affect your Checkmk installation. However, a general statement is unfortunately impossible. In this article we have collected those issues that apply to all or most Checkmk installations. However, there may be other changes that are relevant to you, for example in checks that you use in your installation.

What do you need to do? After you have performed the update, the Checkmk interface will show you the number and content of the incompatible changes and ask you to check and take note of them. So you will definitely be confronted with this issue — but only after you have performed the update.

It is therefore a good idea to get an overview of the incompatible changes before the update. Open the list of Werks. (The software development of Checkmk is organized in so-called Werks). Filter the list of Werks by Checkmk 2.0 and by compatibility, i.e. check the checkbox Only show incompatible. In the description of a Werk you will find hints on what to do, if necessary, to make the change compatible.

The frustrating news: The Werk list of a version is very, very long — even if it only contains the incompatible changes. The comforting news: You can see from the number of changes how much effort we put into improving Checkmk for you.

3. Update

3.1. Best practices for update

In the following we describe best practices which we follow even when updating large Checkmk environments. These are certainly not mandatory in every environment, but they can make the process of updating easier for you.

Updating the operating system

The operating system on a Checkmk server should always be up to date anyway. Before updating Checkmk, however, it certainly does not hurt to make sure that it is up to date, again using apt (for Debian and Ubuntu), yum (for Red Hat and CentOS) or zypper (for SUSE Linux Enterprise Server).

Updating Checkmk version 1.6.0

An update to version 2.0.0 currently does not require a specific 1.6.0 patch release. However, there are good reasons to start the jump to 2.0.0 only from the latest 1.6.0 patch release, because e.g. a 1.6.0 patch may contain fixes that make it easier to update to the 2.0.0 version.

Therefore, we recommend to first update Checkmk to the latest 1.6.0 patch release and only then update to 2.0.0.

Perform a dry run of the update

In large environments, where even restoring a naturally-existing backup of your Checkmk environment would require a certain amount of time, it is advisable to carry out a test with a cloned site before updating the production environment. For this purpose, you can, for example, restore the last regular backup of your site under a different name.

root@linux# omd restore newsite /path/to/backup

Alternatively you can also copy your site via omd cp. For this, however, the site must be stopped for a short time:

root@linux# omd stop mysite
root@linux# omd cp mysite newsite

Afterwards, run the update on this new cloned site to check the local changes mentioned above in the new environment.

Temporarily disable Agent Updater

CEE If you use the automatic agent updates, you should consider disabling this temporarily before updating to Checkmk 2.0.0. Three clicks on Monitoring Agents > Automatic updates and the button edit pen before Master Switch will take care of this.

update 20 automatic agent updates

After the successful update to Checkmk 2.0.0 you can reactivate the updater under Setup > Agents > Windows, Linux, Solaris, AIX > Agents > Automatic updates.

We recommend to reactivate the automatic agent update for a select few of your hosts or host groups at first. In this way, the new agent is not immediately rolled out across all of your servers and you can familiarize yourself with the newly-delivered data on a few systems. Also, due to the significantly-increased number of check plug-ins supplied, you could find a whole range of new services which you can then set up correctly on the test systems you have chosen. New threshold values may also be required for new services. If you tackle this on a small scale first, you may save yourself some false alarms later.

On the page mentioned above you can simply add a few hosts or host groups to the corresponding field and reactivate the master switch.

update 20 activate update on selected hosts

Just remember to remove these restrictions to explicit hosts and host groups once you are done testing the new agent and are satisfied with the results.

3.2. Updates in distributed monitoring

There are different procedures, to perform the update: All in one, successive with disabled configuration changes and completely during the operation. You have to create backups in each case.

The recommended and safe procedure for updating all monitoring instances is as follows:

  • Stop all instances.

  • Execute the updates.

  • Start the updated instances again.

Successive procedure

The standard procedure demands some time and a well planning for bigger environments — the monitoring will be offline for quite some time. Alternatively you are able to prohibit every change in configurations by a global setting. In this case only explicitly marked users are allowed to perform changes so you may update all instances bit by bit. To activate the Read-Only-Mode, you navigate in the central 1.6.0 instance to WATO > Global Settings and click on button readonly:

update 20 readonly

You can activate the mode permanently or for a certain period of time and attach an additional explanation. As long as the mode is active, all users will see the following warning in all settings dialogs — including your explanation:

update 20 readonly warning

The hint But you are allowed to make changes anyway. is of course only displayed for users that have been added to the exception list.

Update during operation

There is a third option: Updating during full operation. You may need this option for example if you manage a bigger amount of instances and the first two options are not suitable. this may be the case for technical or organizational reasons.

This procedure is not recommended and performed at your own risk!

Operating with different versions gives you the basics functionality. But some features of Checkmk may work in this case with restrictions or even not at all. Before you decide to use this procedure, you need to test the update for you individual set of features. In this scenario it is even more important that you take care, that your backup is running and working, because you have to expect bigger problems. You will avoid such problems at best if you update the central instance as last of all.

Please also notice the hints in the Outlook regarding the compatibility mode.

3.3. Carry out the update

Nothing fundamental has changed in the actual update of the software in Checkmk 2.0.0, i.e. you install the new version, perform the update of the Checkmk site, take care of any possible conflicts, and check and confirm the incompatible changes.

Carry out the update procedure as described in the article on updates and upgrades.

4. Follow-ups

4.1. The new user interface

The not to be overlooked innovation in the Checkmk version 2.0.0 is the completely-redesigned user interface (GUI) that presents itself after you have logged in:

gui

The following overview is intended for the version switchers in a hurry and shows where the most important of the well-known GUI elements from version 1.6.0 are located in version 2.0.0:

BeforeNowNote

-

Navigation bar

The new navigation bar is always on the left and offers new menus, most importantly: Setup and Monitor.

Sidebar

Sidebar

The sidebar is on the right-hand side by default and shows important status information for the operator. The sidebar can be extended with snapins as always.

Sidebar > Checkmk logo bar with information on the Checkmk edition, version, number of incompatible changes and a link to the release notes

Help menu

The Help menu of the navigation bar displays the Checkmk edition and version. The number of incompatible changes after an update is displayed in the icon of the Help menu, if applicable. There are Help menu items for calling up the release notes.

WATO - Configuration

Setup menu

The Setup menu in the navigation bar replaces the WATO - Configuration snapin in the sidebar. For the sidebar there are the two snapins Setup and Quick Setup.

WATO > Host & Service Parameters

Setup menu

The rule set categories can be found, completely revised, in the Setup menu under Hosts, Services and Agents.

WATO > Monitoring Agents

Setup > Agents > Windows, Linux, Solaris, AIX

Access to the agent bakery in the CEE Checkmk Enterprise Editions

-

Search field in Setup menu

New is the search in the configuration environment (among others for rule sets and global settings).

Views

Monitor menu

The Monitor menu in the navigation bar replaces the Views snapin in the sidebar, but this is still available.

Quicksearch

Search field in Monitor menu

The Quicksearch snapin for the sidebar is still available.

Main page > Header with user name, time and book symbol for displaying the help texts

Main page > Header with number of changes not yet activated

The current user name with its assigned role is displayed in the User menu of the navigation bar. The help texts can be shown and hidden again on each main page in the Help menu under Context sensitive help.

Main page with buttons

Main page with breadcrumb, menu bar and action bar

The 'breadcrumb' navigation shows the path to the current page, followed by the context-specific menu bar and, again below this, an action bar with buttons for the most important menu items.

In the Beginner’s guide you will find a detailed introduction to the most important elements of the new GUI.

4.2. Fetcher and checker

In Checkmk 2.0.0 the principle of the auxiliary processes has changed a lot. Up to Checkmk 1.6.0 the Checkmk helpers were responsible for performing checks. The Checkmk helper processes remained permanent and thus provided a significant improvement in performance because the time-consuming creation of new processes could be dispensed with.

In Checkmk 2.0.0, these auxiliary processes perform their tasks in an even more specialized and targeted way. Instead of fetching the information and subsequently evaluating it, as in 1.6.0, one part of the auxiliary processes is only responsible for fetching the information. These processes are called Checkmk fetchers. The other part, on the other hand, takes care of evaluating the information passed by the fetchers. We call these processes Checkmk checkers.

Does this affect you? In short, yes — if you are using the CEE Checkmk Enterprise Editions in which the above processes are available. Every administrator of the Checkmk 2.0.0 Enterprise Editions should deal with this point in the Global settings.

What do you need to do? You need to go to Setup > General > Global settings and adjust the values for Maximum concurrent Checkmk fetchers and Maximum concurrent Checkmk checkers.

For the checkers, it can be clearly said that the value should in no case be greater than your server can process in parallel. As a rule, this number corresponds to the number of processor cores in your Checkmk server. However, this would be the absolute maximum, and exceeding it makes no sense.

In the case of the fetchers, however, the available working memory of the server is the limiting factor. The number of Maximum concurrent Check_MK checks from Checkmk 1.6.0 is taken as the initial value here. To find a good setting for this, we recommend activating the Core statistics snapin of the sidebar. This shows you the percentage utilization of the fetchers and checkers. If the fetchers here are regularly close to 100%, you should increase their number in order to obtain a high-performance system. We recommend setting the fetchers so that their load is at 80% a few minutes after restarting a site.

Detailed information can be found here.

4.3. Check automigration to the new Check API

You can check if all local plug-ins from the directories ~/local/share/check_mk/checks and ~/local/share/check_mk/inventory can be automatically migrated to the new Check API.

Does this affect you? This check is useful if you have checks in your local files that were programmed conforming to the old Check API that was valid up to version 1.6.0.

What do you need to do? As a site user, run the following command:

OMD[mysite]:~$ cmk -R
Failed to auto-migrate legacy plugin to check plugin: mssql_counters
Please refer to Werk 10601 for more information.

The message above shows that a plug-in could not be migrated automatically. All of the listed plug-ins will need to be migrated manually by you. There are a number of reasons why an automigration cannot be performed, and you can read about these in the mentioned Werk #10601. You can find out how to carry out manual migration in the Blogpost on the migration of check plug-ins.

4.4. Discover new services

Checkmk 2.0.0 brings with it a whole series of new check plug-ins. If you do not use the automatic update of the service configuration via the periodic service discovery, you will have to search for services on a number of hosts.

If your hosts are organized accordingly (e.g. in folders), you can usually use the Bulk discovery function for this. You can find this under Setup > Hosts > Hosts and then in the menu Hosts > Discover services.

Below we list some components with the most new features.

MySQL server

Especially on servers running MySQL, Checkmk will offer you a whole range of new services for your monitoring.

Network devices

In the past, special characters have crept into the names of network devices in many IT landscapes. Checkmk 2.0.0 will handle these special characters in a completely different way in order to reduce problems. Regardless of whether such naming is good and sensible, a service discovery must be started on the corresponding devices.

Kernel services

The three services Kernel Process Creations, Kernel Major Page Faults and Kernel Context Switches are combined in Checkmk 2.0.0 into one service called Kernel Performance. On the Checkmk servers themselves, a new service discovery must therefore also be triggered. This should of course be done on each site. On the command line you can do this with the following command:

OMD[mysite]:~$ cmk -IIv --plugins kernel,kernel_performance

5. Diagnostics

5.1. Error message: inconsistent use of tabs

Python 3 is much stricter than its predecessor version — also with regard to the requirements for correct syntax. In the beta phase of Checkmk 2.0.0, we repeatedly encountered plug-ins in which even small errors could hinder a successful update. If the error 'inconsistent use of tabs' occurs during the update or the subsequent start of the site, we recommend removing the affected plug-in with the following command.

OMD[mysite]:~$ mkp remove NameOfPlugin

6. Outlook

This chapter deals with topics that do not concern the current Checkmk version 2.0.0, but one of the subsequent versions.

6.1. Distributed monitoring with different Checkmk versions

The synchronization of configuration files between sites in distributed monitoring has changed in version 2.0.0. In version 1.6.0 all configuration files are synchronized between the central site and the remote sites; in version 2.0.0 this happens incrementally, so that only those files that have changed will be synchronized.

To enable the updating of all sites utilizing the distributed monitoring to version 2.0.0 during operation, the synchronization will also work for a transitional period in a mixed environment, as the incremental synchronization is performed only if both instances are running on version 2.0.0. Otherwise the old method will be used. Please read absolutely the hints regarding the Update during operation.

This compatibility mode will be removed for the next Checkmk version (currently planned as 2.1.0). We recommend that all sites using distributed monitoring be updated to version 2.0.0 as soon as possible.

6.2. Discontinuation of the Web-API

The Web-API (also called the HTTP-API) will no longer be supported in the next but one Checkmk version (currently planned as 2.2.0). The Web-API will be replaced by the new REST-API introduced in 2.0.0.

On this page