 |
The functionality presented here is a technical preview, i.e. a preview of a new feature that will be subject to development and expansion until further notice. During this phase, it is possible that functionality will not only be added, but also modified in such a way that existing configurations will become obsolete and you will have to recreate them. We ask for your understanding in this matter. |
1. Introduction
Starting with Checkmk 2.4.0 you can receive OpenTelemetry metrics from 
Checkmk Cloud onwards, i.e. in Checkmk Cloud and Checkmk MSP, and process them in the monitoring.
For this purpose, Checkmk comes with an OpenTelemetry collector.
This collector supports the gRPC and HTTP(S) transport protocols as recipients in a push configuration.
As a scraper, it can also collect data from Prometheus endpoints (in a pull configuration).
Based on any OpenTelemetry services identified, hosts can be created automatically with the dynamic host management. These hosts are then assigned OpenTelemetry metrics as Checkmk services by a special agent.
Briefly summarized: Three components (collector, dynamic host management and special agent) must interact before services can be discovered and metrics generated. You should therefore configure these three components first and only activate the pending changes at the end.
|
This article describes the status of OpenTelemetry integration at the time of release of Checkmk 2.4.0p10. We are continuing to work on incorporating subsequent changes from patch releases. Some of the limitations described have already been fixed and frequently requested features have been added. Please refer to the OpenTelemetry related Werks to find out the current status for the Checkmk version you are using. |
2. Setting up
2.1. Activating the OpenTelemetry collector
First, you must activate the collector when the site is stopped.
This can be done as the site user on the command line using omd config in the text-based configuration menu:
OMD[mysite]:~$ omd configGo to Addons and activate OPENTELEMETRY_COLLECTOR:
┌──────────────────────────Addons─────────────────────────────┐ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ MKEVENTD on │ │ │ │ MKEVENTD_SNMPTRAP off │ │ │ │ MKEVENTD_SYSLOG off │ │ │ │ MKEVENTD_SYSLOG_TCP off │ │ │ │ OPENTELEMETRY_COLLECTOR on │ │ │ │ OPENTELEMETRY_COLLECTOR_SELF_MONITORING_PORT 14317 │ │ │ │ TRACE_RECEIVE off │ │ │ │ TRACE_SEND off │ │ │ │ │ │ │ └─────────────────────────────────────────────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ < Change > <Main menu> │ └─────────────────────────────────────────────────────────────┘
You do not need to change the port for the self-monitoring of the collector, as it is set to the next free port from 14317 even if multiple sites are being used.
After activating the collector, start the site again with omd start.
2.2. Setting up the OpenTelemetry collector
Call Setup > Hosts > OpenTelemetry collector (experimental) and start setting up a new collector with the Add OpenTelemetry collector configuration button:
In General properties, assign an ID and a title for the new collector.
For Site restriction you must move at least one entry from the left-hand side to the right-hand side.
In the image above, the local site mysite is the only one available.
Pay attention to the following points in the collector’s properties:
You must set up at least one of the two push endpoints (via gRPC or HTTP) or a Prometheus scraper (pull). The picture shows an unencrypted, unauthenticated gRPC endpoint on the standard port 4317 on all IPv4 addresses (
0.0.0.0).Create at least one field for the Host name computation. In the example, the simplest option has been selected: the
service.namefield received via OpenTelemetry is used as the host name in Checkmk. In the inline help you will find more information on how to create host names from the OpenTelemetry data.
Use Save to save the collector.
2.3. Setting up dynamic host management
To ensure that hosts are created automatically, set up a new OpenTelemetry connection with the dynamic host management. In preparation — at least for the first tests — we recommend creating a separate folder in which you store the automatically-generated OpenTelemetry hosts.
To set up the new connection for the dynamic host management, go to Setup > Hosts > Dynamic host management > Add connection:
Make the following settings in the Connection properties box:
Set Connector type to OpenTelemetry collector data.
The
OpenTelemetryTestfolder is used as the target folder for the new hosts (Create hosts in) in the example.Set the Host attributes according to your system environment. Unless you have taken the trouble to create names of existing hosts when setting up the OpenTelemetry collector, the automatically-created hosts will be virtual hosts that are only used in the context of OpenTelemetry. You can therefore only use special agents as monitoring agents (Configured API integrations, no Checkmk agent) and leave the settings for IP addresses at No IP.
By activating the checkbox at Service discovery, you specify that service discovery will be performed on the automatically-generated hosts. This leads to the desired result if the special agent for OpenTelemetry has been configured and is active.
For the other parameters, you can find more information in the article on dynamic host management.
Complete the new connection with Save.
2.4. Configuring the special agent
To ensure that Checkmk services are also generated, at least one configuration of the OpenTelemetry special agent must be created. The required rule can be found under Setup > Agents > Other integrations > OpenTelemetry (experimental):
In the OpenTelemetry (experimental) box, decide whether you want to monitor the OpenTelemetry collector itself — via the port that was displayed during activation of the collector.
In the condition for the rule, it is obvious to assign the special agent to the folder in which the hosts will be automatically-created — in the example, this is the OpenTelemetryTest folder.
Save the rule with Save.
Now that the three components — collector, dynamic host management and special agent — have been configured you should activate the pending changes.
2.5. Sending test data to the collector
You will probably want to set up the OpenTelemetry collector for Checkmk because somewhere in your IT environment something is already generating some OpenTelemetry metrics.
If this is not yet the case, or if you want to quickly try out the example setup described in this article, you can do so with the example application "Hello metric" provided in our GitHub repository, which runs in a virtual Python environment.
It takes around two to three minutes from the time the first data is sent, the host has been created and the service has been detected, and then becomes visible in the monitoring. The following image shows the services of the "Hello metric" sample application:
The host name hello-metric is generated from the service name given as an option in the opentelemetry-instrument command — exactly as specified in the configuration of the OpenTelemetry collector.
The OTel metric hellolevel service name in Checkmk is generated from the name of the metric, with the OTel metric prefix added by the special agent.
If you activate the Monitor the collector option in the special agent configuration, i.e. the self-monitoring of the collector, a number of additional services will be created at the host,
|
Only OpenTelemetry metrics that contain exactly one data point are displayed as a service with exactly one metric in Checkmk. |
The "Dice Roll" sample application presented on the OpenTelemetry website works in a very similar way to "Hello metric".
You can use the virtual Python environment created for the "Hello metric" example to try out the "Dice Roll" example.
However, you must also install the Python flask module (pip install flask) and create the app.py file in the metrics-enhanced version, as described here.
In the following call, replace the Checkmk server as the target (here 198.51.100.42 with gRPC port 4317):
user@host:~$ opentelemetry-instrument \
--traces_exporter console \
--metrics_exporter otlp \
--exporter_otlp_metrics_endpoint http://198.51.100.42:4317 \
--logs_exporter console \
--service_name dice-server \
flask run -p 8080After a few minutes, the host dice-server will appear in the monitoring with the service OTel metric dice.rolls. In the "Dice Roll" example application, however, the ‘dice.rolls’ metric is nested and therefore only generates an informative service that always remains OK. Instead of one nested metric, create two ‘flat’ metrics to get two separate services in Checkmk.
2.6. Customizing services
You can define threshold values and more for the services using the Setup > Services > Service monitoring rules > OpenTelemetry (experimental) rule.
3. Files and directories
| File path | Description |
|---|---|
|
This is where OpenTelemetry data is created. A subdirectory is created for each host. The files created there are in JSON format. |
|
Log file of the OpenTelemetry collector |
|
Temporary directory in which aggregated data from the collector is stored for evaluation by the special agent. |
|
Temporary directory where files required by the suggestion function (list entries and suggestions when typing) are stored. |