Checkmk
to checkmk.com

1. Introduction

Should you ever be confronted with problems in Checkmk that you — with the help of this User guide — cannot solve by yourself, the official Checkmk-Support and the Checkmk-Forum are available which provide two excellent contact points. For both a precise description of the error or problem is naturally essential. In addition, certain specific information about your Checkmk environment is mandatory for a quick analysis and solution. The most essential information is of course the version and the edition of Checkmk that you are using. Depending on the situation, however, considerably more information may be required to be able to get to the bottom of your problem. In the past Checkmk support told you what information you needed to provide them with.

But under version 2.0.0 life can now be much easier as you have a tool that generates the required information — the Support diagnostics. Instead of the support providing you with a 'wish list' as mentioned above, and thus sending you on a scavenger hunt through the various components of Checkmk, with just a few clicks in the graphical user interface you can create a tailor-made package of information — referred to as the dump in this article. Here you decide for yourself whether you want to include configuration files that may possibly contain confidential information — or not, and if so to specify which ones.

CEE Some of the selectable data is only available in the commercial editions, for example Performance Graphs of Checkmk Server, CMC (Checkmk Microcore) and Licensing information.

2. Compile support information

2.1. Select site

After clicking on Setup > Maintenance > Support diagnostics you will see the following tidy screen:

Support diagnostics options.

If you are using distributed monitoring, in the first field you can specify the Site from which you want to select the data to be collected.

2.2. General information

Under General information you will basically only find a notice that the information on the Checkmk version and the operating system in use will in any case be included. If you leave it at that — i.e. select none of the options — and start the collection of the data via Collect diagnostics, you will get a file that contains only the following data:

general.json
{
    "arch": "x86_64",
    "core": "cmc",
    "edition": "cee",
    "os": "Ubuntu 22.04.2 LTS",
    "python_paths": [
        "/opt/omd/versions/2.2.0p1.cee/bin",
        "/omd/sites/mysite/local/lib/python3",
        ...
        "/omd/sites/mysite/lib/python3"
    ],
    "python_version": "3.11.2 (main, Mar 14 2023, 20:27:12) [GCC 12.2.0]",
    "time": 1681821714.8444054,
    "time_human_readable": "2023-04-18 14:41:54.844405",
    "version": "2.2.0p1"
}

In addition to the general.json file, Checkmk generates further files in this standard setting with performance data such as the number of hosts and services, the hardware of the Checkmk server, the environment variables set in the site, the names and sizes of the files in the site and the Python modules installed there with their versions. The Background job details page, which is opened after you have clicked Collect diagnostics, shows exactly which data is compiled.

2.3. Optional general information

Under Optional general information you will then find choices which you can compile in advance to suit your question, or which can be or which may be explicitly requested by the support personnel.

If you select Local Files and MKPs here, Checkmk will also create an overview of all files in your site’s ~/local/ directory. This can sometimes be helpful if your local customizations are incompatible with a recent update of Checkmk. All installed MKPs are also included here.

With the selection of OMD Config you can add the information on your configuration of OMD to the dump. These correspond exactly to what you would get on the command line with the omd config show command.

omd_config.json
{
    "CONFIG_ADMIN_MAIL": "",
    "CONFIG_AGENT_RECEIVER": "on",
    "CONFIG_AGENT_RECEIVER_PORT": "8000",
    "CONFIG_APACHE_MODE": "own",
    "CONFIG_APACHE_TCP_ADDR": "127.0.0.1",
    "CONFIG_APACHE_TCP_PORT": "5000",
    "CONFIG_AUTOSTART": "off",
    "CONFIG_CORE": "cmc",
    "CONFIG_LIVEPROXYD": "on",
    "CONFIG_LIVESTATUS_TCP": "off",
    "CONFIG_LIVESTATUS_TCP_ONLY_FROM": "0.0.0.0 ::/0",
    "CONFIG_LIVESTATUS_TCP_PORT": "6557",
    "CONFIG_LIVESTATUS_TCP_TLS": "on",
    "CONFIG_MKEVENTD": "on",
    "CONFIG_MKEVENTD_SNMPTRAP": "off",
    "CONFIG_MKEVENTD_SYSLOG": "on",
    "CONFIG_MKEVENTD_SYSLOG_TCP": "off",
    "CONFIG_MULTISITE_AUTHORISATION": "on",
    "CONFIG_MULTISITE_COOKIE_AUTH": "on",
    "CONFIG_PNP4NAGIOS": "on",
    "CONFIG_TMPFS": "on"
}

If you check the Checkmk Overview checkbox, general information about all sites running on your Checkmk server will be included. This also creates a list of all installed Checkmk versions. And if the site selected above is the node for a cluster this fact will also be recorded here.

With the Crash Reports option, you can add the latest crash reports from the various categories — e.g. base, check, gui or rest_api — to the dump. Further information on crash reports can be found in the article on programming agent-based check plug-ins. In Checkmk you can find the crash reports under Monitor > System > Crash reports.

After activating Checkmk Log files or Checkmk Configuration files you have the option of removing any confidential data from this part of the package — via the Pack only Low sensitivity files selection. All files you can select here come from the ~/var/log/ or ~/etc/checkmk/ directories and their subdirectories. You can see a detailed list directly below the drop-down menu. With the Select individual files from list option you even have the option to include only specified files into the dump.

With this option you can also see which files have one of the confidentiality levels High (H), e.g. files with passwords, Medium (M), if they contain addresses or user names for example, or finally Low (L).

List of selectable files.

Note: In order to have as much detailed information as possible on the behavior of Checkmk in the log files, it may be necessary to change the log level in Checkmk for a short time. The corresponding settings can be found via Setup > General > Global settings. On this page, simply type logging into the Filter field and then set, for example, the log level for the Core to Debug. If you now simply let the site continue to run for a few minutes or if you repeat a reproducible error, the chances increase that information on this error show up in the log files.

Next, you have the option to add Performance Graphs of Checkmk Server to the dump. Especially in cases of problems with the performance of a Checkmk site these reports are almost always requested — so it is a good idea to include with such problems. The Support diagnostics will do the work for you to manually generate a number of reports as PDF files. Among others, the reports from the OMD mysite performance service for the last 25 hours and the last 35 days are generated.

2.4. Component-specific information

The Component specific information section again allows you to decide in a very granular way on which information from your global Checkmk settings, your hosts and folders, and your notification settings should be included in the dump.

Important: Depending on the configuration, the files you select here may contain confidential information, such as passwords. In normal operation, this data is protected by the fact that only the site user and administrators have access to it. If you make this data available to third parties for analysis purposes, you should proceed with great caution.

In the Global Settings sub-item you will find all global.mk-files of the individual components of your Checkmk site, such as the dynamic configuration daemon or the live status proxy daemon.

The information that can be selected via the sub-item Hosts and Folders can, among other things, help to find unfavorable rule sets and errors in the host configuration.

In the Notifications section you will find, in addition to the corresponding configuration files, an option for the selection of log files. In case of difficulties with your notifications, you — or ultimately the Checkmk support — can often find the source of the problem in these logs.

Selection of files for the notifications.

Use the Business Intelligence option to select the configuration files for Business Intelligence. After activating one of the options CMC (Checkmk Microcore) or Licensing information you again have the possibility to remove more or less confidential data from the respective part of the package — via the Pack only Medium and Low sensitivity files or Pack only Low sensitivity files selection.

3. Support diagnostics via the command line

As is often the case in Checkmk, this task can be performed simply from a terminal. This can be done easily with the cmk command and the --create-diagnostics-dump option. For all options described above you can append the corresponding parameter to the command.

OMD[mysite]:~$ cmk --create-diagnostics-dump --local-files --omd-config --performance-graphs

The following options complete the output from the command:

--local-files

List of all installed, unpackaged, and optional files below ~/local/. This also includes information about installed MKPs.

--omd-config

Contents of the ~/etc/omd/site.conf file.

--checkmk-overview

Information from HW/SW inventory node Software > Applications > Checkmk of the Checkmk server.

--checkmk-config-files FILE,FILE …​

All .mk and .conf files from the ~/etc/checkmk directory.

--checkmk-log-files FILE,FILE …​

All log files (.log and .state) from ~/var/log.

--performance-graphs

Performance graphs (e.g. CPU load, CPU utilization) of the Checkmk server — only available in the commercial editions.

The above, and all other options for the cmk command can as usual be found in the output from cmk --help.

4. Missing information in the dump

4.1. Current agent needed

To be able to output specific information in the Support diagnostics, you must make sure that the Checkmk servers have the agent with the version number of at least 2.0.0 installed. In particular, the information that comes from the HW/SW inventory of the Checkmk server was not provided at all in older versions of the agent.

4.2. Label cmk/check_mk_server:yes

The Support diagnostics depend on the Checkmk servers in your environment being labeled accordingly. If you are missing some data in a dump, check if your Checkmk servers are labeled with the cmk/check_mk_server:yes label.

On this page