Checkmk
to checkmk.com

1. Introduction

grafana logo

Checkmk has an integrated graphing system with comprehensive features for visualizing and storing of metrics. However it might still be helpful to use Grafana as an external graphing system — for example, if you are already using Grafana and have other data sources connected to it and want to have a single, unified dashboard.

Beginning with version 2.0.0, it is possible in all Checkmk editions to directly address Checkmk as a data source in Grafana 8.x, and to display individual metrics – or even entire graphs as predefined by Checkmk – in Grafana. In addition you can create your own graphs dynamically using regular expressions to specify a set of hosts and services that should take specific metrics in the graph into account.

How to get and to display your Checkmk metrics in Grafana is explained fundamentally in this article. For detailed instructions on how to use and configure Grafana, see the Documentation at Grafana Labs.

Since the plug-in for Grafana is developed in parallel to Checkmk, it is not included with Checkmk. You can however keep up-to-date with the latest development status through the Archive on GitHub.

An integration directly into Grafana is currently still being tested. Once the plug-in has been officially incorporated you will be able to install it directly from Grafana’s interface.

2. Setting-up the plug-in in Grafana

2.1. Installing the plug-ins

To use the plug-in, there are two ways to add it into Grafana:

  • You download the ZIP file from the GitHub project, and manually paste the content into the Grafana plug-in directory.

  • You clone the GitHub project directly into the plug-in directory.

Installation from the ZIP file

This variant is the simplest and is preferred if you have not installed git on the Grafana server and cannot/do not want to. To install the plug-in, simply download the ZIP file with the latest version, and copy it – for example, with scp – to the Grafana server:

grafana download plugin

Alternatively, you can also load the file directly from the command line. Note that you need to know the correct version for this. With the option -O in the following comannd, the name of the file to be saved locally is set manually. Otherwise it would be 2.0b2.zip:

root@linux# wget -O grafana-checkmk-datasource-2.0b2.zip https://github.com/tribe29/grafana-checkmk-datasource/archive/refs/tags/2.0b2.zip

The content is then extracted into the Grafana plug-in directory. This is usually the file path /var/lib/grafana/plugins/:

root@linux# unzip grafana-checkmk-datasource-2.0b2.zip && \
mv -v grafana-checkmk-datasource-2.0b2 /var/lib/grafana/plugins/grafana-checkmk-datasource

Finally you can activate and set up the plug-in in the Grafana interface.

Installation by cloning the Git archive

The variant described above has the fewest requirements, and is easy to implement even for less-experienced users. But if you get the plug-in directly from the Git archive, there are several advantages:

  • Upgrading to a new version can be quickly performed with two commands: git pull

  • You have the option of using the current development version directly from the archive if you want to test a new feature: git checkout develop

To use the plug-in with the help of a copy of the archive, you absolutely need the git program. The procedure is then quite simple – simply clone the archive into the Grafana plug-in directory:

root@linux# cd /var/lib/grafana/plugins/ && git clone https://github.com/tribe29/grafana-checkmk-datasource.git
Cloning into 'grafana-checkmk-datasource'...
remote: Enumerating objects: 541, done.
remote: Total 541 (delta 0), reused 0 (delta 0), pack-reused 541
Receiving objects: 100% (541/541), 291.55 KiB | 0 bytes/s, done.
Resolving deltas: 100% (374/374), done.
Checking connectivity... done.

Since the master branch always shows the latest version, after a new release you just need to execute the following command to update the plug-in on the Grafana server:

root@linux# cd /var/lib/grafana/plugins/grafana-checkmk-datasource && git pull

When using the development tree, you need to create missing cache directories, etc. This is done by the following command:

root@linux# cd /var/lib/grafana/plugins/grafana-checkmk-datasource && yarn run build

2.2. Allowing unsigned plug-ins

As of Grafana 8.0, it is no longer possible to easily integrate external plug-ins without further ado as long as they are not signed. This applies to the Checkmk plug-in which is still under development, so that you need an additional step.

To do this, modify the configuration file /etc/grafana/grafana.ini and change the line ;allow_loading_unsigned_plugins = as follows. Note to remove the semicolon at the beginning of the line:

/etc/grafana/grafana.ini
[plugins]
allow_loading_unsigned_plugins = tribe-29-checkmk-datasource

With this setting you only add an exception for this one plug-in. If you are using Grafana in a (Docker) container, either link the custom INI file into the container or create a Volume which you map to the directory in the container (/etc/grafana). The following is an example of how to link the INI file using --mount. Note that this is only an example and will probably not fit your environment directly:

user@host:~$ sudo docker run -d --mount type=bind,source=/home/user/grafana/grafana.ini,target=/etc/grafana/grafana.ini -p 3000:3000 --name grafana -v grafana-plugins:/var/lib/grafana/plugins -v grafana-db:/var/lib/grafana grafana/grafana

2.3. Restarting the Grafana service

After changing the configuration file, updating or installing the plug-in, restart the Grafana server service:

root@linux# service grafana-server restart

Afterwards the plug-in is available in the Grafana GUI, and from there can be activated and set up.

2.4. Activating and setting-up the plug-in

After the necessary files have been installed you can activate the plug-in in Grafana. Go to the configuration and select the Data sources tab. Here you can add a new data source using the Add data sources button:

grafana plugins overview

The entry for Checkmk can be found under the category Others:

grafana plugin cmk

The configuration mask for the data source is quite simple. Enter the URL for your site and a user who has rights to access the Web API. Important: In a distributed setup with multiple sites, here you specify the URL for the central site:

grafana plugin config

If you want to connect more than one Checkmk site, you can optionally add a unique name to each connection — otherwise simply leave the default tribe-29-checkmk-datasource as is.

After you have saved the connection with the Save & Test button, it will be available as a data source in Grafana and you can configure your first graphs.

3. Creating graphs

3.1. Creating a blank dashboard

Dashboards are generated in Grafana using the ‘plus’ icon on the right side. With a click on Dashboard you can create something like this:

grafana new dashboard

3.2. Displaying a single metric from a host

The simplest dashboard shows a single graph of a service of a host. This can also be the metric of a local check for which no prebuilt graphs exist. In an existing dashboard or the one you just created, create a new panel: First select Add Query here:

grafana dashboard addpanel

Creating the panel immediately takes you to adding a new Query. As Data source select tribe-29-checkmk-data-source from the dropdown menu. After that you can still limit the query to one Checkmk site (Site). Now filter further for Hostname and Service. Finally, select the Aggregation (in the example Lines), as Graph type the Single metric type and finally the metric of the service to be displayed (for services containing multiple metrics).

grafana dashboard single config

Grafana will display the result directly. Once you have clicked on the save icon in the top right corner, you will be prompted for a title for the panel. You will then be taken back to the dashboard:

grafana dashboard single view

3.3. Displaying a predefined graph from Checkmk

Developers of Checkmk have the ability to summarize corresponding metrics in a graph to provide an overview of different aspects of a resource. You can display the metrics from such a ready-made graph directly in Grafana.

The procedure is initially the same as explained in the previous section. However, to finish, select as Graph type the Template entry and the Template of the service to be used.

grafana dashboard predefined config

Once you are satisfied with the selection made, click the save icon and specify a title for the panel. The graph is now available in the selected dashboard.

grafana dashboard predefined view

3.4. Creating complex graphs

Especially in a dynamic cluster, you often want to be able to track the entire history of a metric across all participating hosts without having to adjust a graph each time a new node is added or dropped. Therefore, you have several options to dynamically customize graphs.

The first way is to filter by Host Labels or Host Tags instead of selecting individual hosts. Another option is to use regular expressions. You can apply regular expressions to hosts or services. In the example, filesystem.* matches all services that start with filesystem — no matter how many file systems a host has mounted.

grafana dashboard combined config

In addition to the advanced filter options, with Aggregation you can specify the representation of the metrics in the graph, and with Graph, which graph should be used as a reference. Note that metrics for a host/service will only be displayed if the host/service also has this selected graph. The example graph looks like this:

grafana dashboard combined view

4. Changes compared to previous versions

4.1. Omitted variables

Earlier beta versions allowed to control the displayed metric names using variables. Here the following variables were allowed:

VariableDescription

$title

The title of the metric as it would be displayed in Checkmk.

$site

The Checkmk site where the host and its metric are monitored.

$host

The host to which the metric is assigned.

$service

The service to which the metric is assigned.

This feature was removed with the 2.0 series of the plug-in because it ultimately duplicates functionality that is part of Grafana.

4.2. Renaming via regular expressions

You can use the Transform tab and the Rename by regex sub-item here to reorder the naming of metrics or suppress information that is not needed. Our following example names Service and Host explicitly and removes all characters before the last slash:

grafana series renaming regex

The variables $1 to $3 here correspond to the strings "captured" in the line above (match groups or capture groups). Grafana also allows for nested match groups. An overview of their possibilities is shown in the corresponding section in regular expressions article.

5. Files and directories

File pathDescription

/var/lib/grafana/plugins/

Grafana checks here for (new) plug-ins. Each plug-in gets its own subdirectory. Save the Checkmk plug-in here.

On this page