Checkmk
DeutschEspañolFrançaisItaliano日本語
Cloud (SaaS) master2.3.02.2.02.1.02.0.01.6.0
to checkmk.com

Special agents Dynamic host management
Tip

The functionality presented here is a technical preview, i.e. a preview of a new feature that will be subject to development and expansion until further notice. During this phase, it is possible that functionality will not only be added, but also modified in such a way that existing configurations will become obsolete and you will have to recreate them. We ask for your understanding in this matter.

1. Introduction

CEE Starting with Checkmk 2.4.0 you can receive OpenTelemetry metrics from CSE Checkmk Cloud onwards, i.e. in Checkmk Cloud and Checkmk MSP, and process them in the monitoring. For this purpose, Checkmk comes with an OpenTelemetry collector. This collector supports the gRPC and HTTP(S) transport protocols as recipients in a push configuration. As a scraper, it can also collect data from Prometheus endpoints (in a pull configuration).

Based on any OpenTelemetry services identified, hosts can be created automatically with the dynamic host management. These hosts are then assigned OpenTelemetry metrics as Checkmk services by a special agent.

Briefly summarized: Three components (collector, dynamic host management and special agent) must interact before services can be discovered and metrics generated. You should therefore configure these three components first and only activate the pending changes at the end.

Tip

This article describes the status of OpenTelemetry integration at the time of release of Checkmk 2.4.0p10. We are continuing to work on incorporating subsequent changes from patch releases. Some of the limitations described have already been fixed and frequently requested features have been added. Please refer to the OpenTelemetry related Werks to find out the current status for the Checkmk version you are using.

2. Setting up

2.1. Activating the OpenTelemetry collector

First, you must activate the collector when the site is stopped. This can be done as the site user on the command line using omd config in the text-based configuration menu:

OMD[mysite]:~$ omd config

Go to Addons and activate OPENTELEMETRY_COLLECTOR:

 ┌──────────────────────────Addons─────────────────────────────┐
 │ ┌─────────────────────────────────────────────────────────┐ │  
 │ │   MKEVENTD                                      on      │ │  
 │ │   MKEVENTD_SNMPTRAP                             off     │ │  
 │ │   MKEVENTD_SYSLOG                               off     │ │  
 │ │   MKEVENTD_SYSLOG_TCP                           off     │ │  
 │ │   OPENTELEMETRY_COLLECTOR                       on      │ │  
 │ │   OPENTELEMETRY_COLLECTOR_SELF_MONITORING_PORT  14317   │ │  
 │ │   TRACE_RECEIVE                                 off     │ │  
 │ │   TRACE_SEND                                    off     │ │  
 │ │                                                         │ │  
 │ └─────────────────────────────────────────────────────────┘ │  
 ├─────────────────────────────────────────────────────────────┤  
 │               < Change  >        <Main menu>                │  
 └─────────────────────────────────────────────────────────────┘

You do not need to change the port for the self-monitoring of the collector, as it is set to the next free port from 14317 even if multiple sites are being used.

After activating the collector, start the site again with omd start.

2.2. Setting up the OpenTelemetry collector

Go to Setup > Hosts > OpenTelemetry collector (experimental) and start setting up a new collector by clicking the Add OpenTelemetry collector configuration button:

“The general properties of the OpenTelemetry collector.”

In General properties, assign an ID and a title for the new collector. For Site restriction, you must move at least one entry from the left to the right side. In the above screenshot, the local site mysite is the only one available.

“The specific properties of the OpenTelemetry collector.”

Basic settings

Pay attention to the following points in the collector properties:

  • You must set up at least one of the two push endpoints (via gRPC or HTTP) or a Prometheus scraper (pull). The screenshot shows an unencrypted, unauthenticated gRPC endpoint on the default port 4317 on all IPv4 addresses (0.0.0.0).

  • Create at least one field for the Host name computation. In the example, the simplest option has been selected: the service.name field received via OpenTelemetry is used as the host name in Checkmk. In the inline help, you can find more information about the options for creating host names from OpenTelemetry data.

Tip

If you want to create more complex host names, save the collector configuration and activate the changes without adding a rule for Host name computation. Then ensure that metrics are sent to the collector. If you now add rules for the Host name computation, suggestions for the available fields will be displayed.

Sending logs to the Event Console

OpenTelemetry also provides options for transferring log messages. The collector can be configured to transfer these in syslog format via TCP to any applications that can receive syslog messages. In the context of Checkmk, the processing of log messages by the Event Console is of interest. You can activate this function with the Send log messages to event console checkbox.

Tip

Note that OpenTelemetry Instrumentation already ensures that log messages are sent sparingly. If it is unclear what is arriving at the Checkmk server, temporarily activate a local rsyslogd that listens on port 514/TCP. This will give you an idea of the incoming data volumes before you proceed with activating the Event Console.

Use Save to save the collector.

2.3. Setting up dynamic host management

To ensure that hosts are created automatically, set up a new OpenTelemetry connection with the dynamic host management. In preparation — at least for the first tests — we recommend creating a separate folder in which you store the automatically-generated OpenTelemetry hosts.

To set up the new connection for the dynamic host management, go to Setup > Hosts > Dynamic host management > Add connection:

Properties of an OpenTelemetry connection in the dynamic host management.

Make the following settings in the Connection properties box:

  • Set Connector type to OpenTelemetry collector data.

  • The OpenTelemetryTest folder is used as the target folder for the new hosts (Create hosts in) in the example.

  • Set the Host attributes according to your system environment.
    Unless you have taken the trouble to create names of existing hosts when setting up the OpenTelemetry collector, the automatically-created hosts will be virtual hosts that are only used in the context of OpenTelemetry. You can therefore only use special agents as monitoring agents (Configured API integrations, no Checkmk agent) and leave the settings for IP addresses at No IP.

  • By activating the checkbox at Service discovery, you specify that service discovery will be performed on the automatically-generated hosts. This leads to the desired result if the special agent for OpenTelemetry has been configured and is active.

  • For the other parameters, you can find more information in the article on Dynamic host management.

Complete the new connection with Save.

2.4. Configuring the special agent

To ensure that Checkmk services are also generated, at least one configuration of the OpenTelemetry special agent must be created. The required rule can be found under Setup > Agents > Other integrations > OpenTelemetry (experimental):

The rule for configuring the OpenTelemetry special agent.

In the OpenTelemetry (experimental) box, decide whether you want to monitor the OpenTelemetry collector itself — via the port that was displayed during activation of the collector.

In the condition for the rule, it is obvious to assign the special agent to the folder in which the hosts will be automatically-created — in the example, this is the OpenTelemetryTest folder.

Save the rule with Save.

Now that the three components — collector, dynamic host management and special agent — have been configured you should activate the pending changes.

2.5. Sending test data to the collector

You will probably want to set up the OpenTelemetry collector for Checkmk because somewhere in your IT environment something is already generating some OpenTelemetry metrics.

If this is not yet the case, or if you want to quickly try out the example setup described in this article, you can do so with the example application "Hello metric" provided in our GitHub repository, which runs in a virtual Python environment.

It takes around two to three minutes from the time the first data is sent, the host has been created and the service has been detected, and then becomes visible in the monitoring. The following image shows the services of the "Hello metric" sample application:

The OpenTelemetry service created with the 'Hello metric' sample application.

The host name hello-metric is generated from the service name given as an option in the opentelemetry-instrument command — exactly as specified in the configuration of the OpenTelemetry collector. The OTel metric hellolevel service name in Checkmk is generated from the name of the metric, with the OTel metric prefix added by the special agent.

If you activate the Monitor the collector option in the special agent configuration, i.e. the self-monitoring of the collector, a number of additional services will be created at the host,

The sample application 'Dice Roll' presented on the OpenTelemetry website works in much the same way as 'Hello metric'. This sample application is in many respects more practical than 'Hello metric'. For example, 'Dice Roll' sends OpenTelemetry metrics every time the application server is called (and only then!) and enters the data type 'Counter'. You can use the virtual Python environment created for the 'Hello metric' example to try out the 'Dice Roll' example as well. However, you will also need to install the flask Python module (pip install flask) and create the app.py file in the version extended with metrics, as described in this instruction.

In the following call, replace the Checkmk server as the target (here 198.51.100.42 with gRPC port 4317):

user@host:~$ opentelemetry-instrument \
    --traces_exporter console \
    --metrics_exporter otlp \
    --exporter_otlp_metrics_endpoint http://198.51.100.42:4317 \
    --logs_exporter console \
    --service_name dice-server \
    flask run -p 8080
Tip

The restriction to exactly one data point that applied up to Checkmk 2.4.0p3 has been removed. Werk #18209 describes how Checkmk metric names are generated for OpenTelemetry metrics with multiple data points.

Your flask server is now ready and can be accessed at the selected port (here: 8080) to roll a die for each call. You can see the die data, for example, as output from curl -v http://localhost:8080/rolldice. Each die roll you trigger in this way is transmitted to Checkmk as a data point.

After a few minutes, the host dice-server will appear in the monitoring with the OTel metric dice.rolls service. This has six graphs, Value_1__Per_Sec to Value_6__Per_Sec, which correspond to the rates of the numbers one to six rolled so far. The reason for the conversion is that OpenTelemetry metrics use counters, which can only increment — a data type that is not provided for in Checkmk. For a clearer display, Checkmk therefore performs the conversion taking the time component into account.

The screenshot shows one of these graphs, which is based on the once per second calls to /rolldice within the monitoring period.

“Graph showing the rate at which the number 5 is rolled.”
OpenTelemetry metrics use single and double underscores, and units are appended to the metric name if provided (here: per sec, i.e., per second).

2.6. Customizing services

You can set thresholds and more for services using the Setup > Services > Service monitoring rules > OpenTelemetry (experimental) rule. Here you can set rules for individual or all metrics of a service. The example screenshot shows the selection of the counter for throws with the number 5, as shown above. The names of known metrics are accessible via the list or as suggestions while typing. You can enter metrics that are not yet known but that you know will appear as free text.

Depending on the data type provided by the OpenTelemetry data point, it may be useful to calculate rates. This is useful for counters, such as for dropped network packets. If multiple sources provide data that is assigned to the same Checkmk service, you can also specify whether aggregation (e.g., sum or average) should be performed or whether simply the most recent, largest, or smallest data points of an interval should have priority.

“Setting the service rules.”

3. Files and directories

File path Description

~/var/check_mk/otel_collector/host_monitoring/

This is where OpenTelemetry data is created. A subdirectory is created for each host. The files created there are in JSON format.

~/var/log/otel-collector.log

Log file of the OpenTelemetry collector

~/tmp/check_mk/otel_collector/self+monitoring/

Temporary directory in which aggregated data from the collector is stored for evaluation by the special agent.

~/tmp/check_mk/otel_collector/autocompleter/

Temporary directory where files required by the suggestion function (list entries and suggestions when typing) are stored.
On this page