Integrating Checkmk in Grafana

This type of installation requires the use of the Checkmk Cloud Edition. If Grafana is operated on its own server, installation from the Grafana plug-in collection is possible via the command line:

Restart the Grafana server service:

The plug-in will then be available in the Grafana interface and so can there be activated and set up.

First log in to the Grafana instance in which you want to install the Checkmk data source. Then open the page for the data source in the Grafana plug-in collection. In the Installation tab, tick the Grafana instance that is to receive the plug-in and then click on the Install button. It may take a few minutes before the plug-in is available in the Grafana installation’s web interface.

On the Releases page in the GitHub repository you will find two variants of the plug-in as a zip archive:

Simply download the latest version as a zip file and copy it, for example with scp, to the Grafana server.

Instead of using the browser, you can of course also load the file directly from the command line. Note that you need to know the current version for this — in the following example, we write the version in the $plugvers variable.

Next, unpack the zip archive:

Create a plug-in directory, if this does not already exist, and make the Linux user under whose ID the Grafana processes are executed (usually grafana) its owner:

Move the resulting folder to the Grafana plug-in directory. This is usually the file path /var/lib/grafana/plugins/:

Change the ownership to the Grafana user:

An installation by cloning the Git archive (as described in older versions of this article) is no longer possible. If you want to participate in the development of the plug-in, read the Development Guide.

Under Home > Dashboards you will find a drop-down menu marked with a plus sign. Click on New dashboard to create a new dashboard:

The simplest dashboard shows a single graph for a host’s service. This can also be the metric from a local check for which no ready-made graphs exist. Create a new visualization in an existing dashboard or the one you have just created. First select Add visualization here:

Creating a visualization takes you to the selection of the data source. After selecting Checkmk, you are directly in the editing mode of the Query for the newly created panel. A panel is a container for a visualization.

Access to the data to be visualized differs slightly between the Checkmk Raw Edition and the commercial editions: In the Raw Edition you work with predefined menus — for site, host name, service and so on. In the commercial editions, you have the option of defining cascades of filters. This allows a very detailed selection, also using regular expressions. The following simple example has been chosen so that you can run through it with all editions.

First select the Site, then filter for a Hostname in the monitoring. For the first test, use any host that uses the Windows or Linux agent. Select Check_MK as the Service. Set the type of Aggregation to Maximum. The Graph type available for selection is Single metric or Predefined graph: Predefined graphs take metrics from Checkmk, which can also contain combined graphs. Our screenshots here show the Single metric, which contains the total execution time of the agent (Time spent waiting for Checkmk agent).

Grafana displays the result directly. With the Apply button you can apply the graph without saving the dashboard. With Save you are prompted to enter a title for the new Dashboard and can then save the newly created dashboard.

The developers of Checkmk have already compiled many related metrics into common graphs to make it easier for users to get an overview of the various aspects of a single resource. You can display the metrics from such a ready-made graph directly in Grafana.

The procedure is initially the same as explained in the previous section. However, to complete the process select the Graph type entry Predefined graph and the Predefined graph for the service to be used.

If you receive an error message that the selection of combined metrics with more than 200 elements is not permitted, use additional filters to restrict the selection. If in doubt, move the filters that will significantly limit the result to the front of the set.

Once you are satisfied with your selection, click the Apply button. The graph will now be available in the selected dashboard.

Especially in a dynamic cluster, you will often want to be able to track the entire progress of a metric across all participating hosts without having to customize a graph every time a new node is added or removed. To achieve this, there are various options for dynamically adapting graphs.

The first option is to filter by Host Labels or Host Tags instead of selecting individual hosts. Another option is to use regular expressions. You can apply regular expressions to hosts or services. In the following example, filesystem matches all services that contain filesystem — regardless of how many file systems a host has mounted.

With filesystem.*nvme you would restrict the selection to all file systems on NVMe drives, no matter which file system type they use, since the part of the string between filesystem and nvme can be arbitrary.

In addition to the extended filter options, use Aggregation to determine the display of the metrics in the graph and Graph to determine which graph is to be used as a reference. Note that metrics for a host/service are only displayed if it also has the selected graph. A graph could then look like this, for example:

In each panel’s edit mode, you will find a Transform tab. You can use the Rename by regex sub-item available here to reorder the names of metrics or suppress information that is not required. Our following example searches for two groups of characters separated by a comma followed by a space, swaps them and explicitly places Service and Host at the front:

The variables $1 and $2 correspond to the strings 'captured' in the line above (match groups or capture groups). Grafana also allows nested match groups. The corresponding section in the article on regular expressions provides an overview of the possibilities available for these.

Each variable is a property of a dashboard. It must be created in the dashboard settings and can then only be used within this dashboard. As query requires the most extensive configuration, we will show its use in our example.

The relevant properties for a variable are its type and its name; all other properties are only used for the purpose of clarity in the dashboard. The name should differentiate between variables. So if you plan to use several variables to select hosts from different subsets, use a name that references this set instead of host as in the example, for example host_from_linuxservers.

When selecting a data source, make sure that a Checkmk connection is selected. You can further restrict the selected object type, for example with regular expressions. Any filters used are evaluated on the Checkmk server. In contrast, the following fields Regex and Sort are evaluated by Grafana.

The two checkboxes for Selection options at the bottom of the page can be optionally used later depending on need. You can modify these settings at any time if required; for the first test, leave both at the (empty) default setting. Save the variables using Save dashboard when you have made all the settings.

You can now enter the assigned variables in the appropriate fields to create a flexible graph. The variables can also be used in the graph’s title.

A drop-down selection for each variable is displayed in the dashboard, which you can use to set the variable values (hosts, services…​) which you want to display on the dashboard. If you have activated multiple selection, you can set the Repeat by variable option in the panel settings to display the panel for each selected variable’s value.