Checkmk
Our AI policyAI

1. Introduction

The Checkmk server typically contacts hosts in pull mode via a TCP connection on port 6556. On Linux and Windows, the Agent Controller typically listens on this port and forwards the agent output via a TLS-encrypted connection. In Checkmk Ultimate, the push mode provides an additional option for selecting the transmission direction.

However, there are environments in which the Agent Controller cannot be used. In legacy mode an inetd is used instead, which executes the agent script for each request, transmits the output in plain text, and then closes the connection. In many cases, security policies require that data not be transmitted in plain text or that existing communication channels be used. While an attacker usually cannot do much with file system usage statistics, process tables or lists of pending security updates can help them prepare targeted attacks.

The universal method for connecting any transport paths to Checkmk is through data source programs. The idea is very simple: You pass a command to the Checkmk server that is used to retrieve agent data. Instead of contacting the host to be monitored on port 6556, Checkmk executes this command. When executing the command, Checkmk expects agent data on the standard output. Further processing proceeds as if the agent data came from a 'normal' agent. Since data sources are treated like agent calls, it is important that you configure hosts that are monitored via data source programs in the Setup GUI with the setting API integrations if configured, else Checkmk agent or Configured API integrations and Checkmk agent.

Ultimately, the plain text output from the agent script can be transferred in any way—directly or indirectly, via pull or push. Here are some examples of how Checkmk users can retrieve data from the agent to the Checkmk server:

  • via email

  • via HTTP access from the server

  • via HTTP upload from the host

  • by accessing a file that was copied from the host to the server using rsync or scp

  • via a script that retrieves the data via HTTP from a web service

  • via an Agent call using an SSH tunnel

When should I use data source programs, and when should I use special agents?

Data source programs and special agents are closely related. Ultimately, special agents can be viewed as an evolution of data source programs. Special agents were initially used to make data source programs more configurable.

Today, special agents typically provide their own agent sections and are combined with their own check plug-ins. In practice, it is important to note that only one data source program can be assigned to a host, but since Checkmk 2.4.0, any number of special agents can be assigned. Special agents can also supplement the regular agent output; a data source program, on the other hand, replaces it.

So, for example, if you can retrieve data via a REST API that is useful for monitoring in addition to the regular agent output, you must use a special agent. If you simply want to transmit regular agent output via other means, data source programs are the right choice. And finally, you can convert any data source program—should greater configurability ever be required—into a special agent.

2. Writing data source programs

2.1. The simplest possible program

The writing and installation of a data source program is not difficult. Any Linux-supported script and program language can be used. The program is best stored in the ~/local/bin/ directory, where it will always be found automatically without the need to specify a data path.

The following first, very basic example is called myds and it generates simple, fictional monitoring data. Instead of integrating a new transport path, it generates the monitoring data itself. These consist of one section <<<df>>>, which contains the information for a single file system, and which has a size of 100 kB and the name My_Disk. It is coded as a shell script of three lines:

~/local/bin/myds
#!/bin/sh
echo '<<<df>>>'
echo 'My_Disk  foobar  100 70 30  70% /my_disk'
Copy file content to clipboard
Successfully copied file content to clipboard!
Write access to clipboard has been denied!

Don’t forget to make the program executable:

OMD[mysite]:~$ chmod +x local/bin/myds
Copy command(s) to clipboard
Successfully copied command(s) to clipboard!
Write access to clipboard has been denied!

Now create a test host in the Setup – e.g., myserver125. This does not require an IP address. In order to avoid Checkmk attempting to resolve myserver125 via DNS, set IP address family to No IP.

Next add a rule in the Setup > Agents > Other integrations > Individual program call instead of agent access rule set which applies to this host, and enter myds as an executable program:

Input mask for an individual command.

When you now go to the host’s service configuration in the Setup GUI, exactly one service ready to start monitoring should be listed:

The new service has been detected.

Add this service into the monitoring, activate the changes, and your first data source program will be running. For a test, as soon as you alter the data being generated by the program the My_Disk file system’s next check will immediately show this.

2.2. Error diagnosis

If something is not functioning correctly, the host’s configuration can be checked by entering cmk -D in the command line and see if your rule has taken effect:

OMD[mysite]:~$ cmk -D myserver125

myserver125
Addresses:              myserver125
Tags:                   [address_family:ip-v4-only], [agent:cmk-agent], [criticality:prod], [ip-v4:ip-v4], [networking:lan], [piggyback:auto-piggyback], [site:mysite], [snmp_ds:no-snmp], [tcp:tcp]
Host groups:            check_mk
Agent mode:             Normal Checkmk agent, or special agent if configured
Type of agent:
Program: myds
Copy command(s) to clipboard
Successfully copied command(s) to clipboard!
Write access to clipboard has been denied!

With a cmk -d you can trigger the retrieval of the agent data as well as the execution of your program:

OMD[mysite]:~$ cmk -d myserver125
<<<df>>>
My_Disk  foobar  100 70 30  70% /my_disk
Copy command(s) to clipboard
Successfully copied command(s) to clipboard!
Write access to clipboard has been denied!

A duplicated -v should generate a message that your program will be invoked:

OMD[mysite]:~$ cmk -vvd myserver125
Calling: myds
<<<df>>>
My_Disk  foobar  100 70 30  70% /my_disk
Copy command(s) to clipboard
Successfully copied command(s) to clipboard!
Write access to clipboard has been denied!

2.3. Transferring a host’s name

The program in our example actually works, but is not very useful as it always produces the same data, regardless of which host it is invoked for.

A real program that, for example, retrieves data via HTTP from somewhere, requires at least the name of the host from where it should retrieve the data. By coding $HOSTNAME$ as a placeholder in the command line you can allow this to be transferred:

Passing the host name with the $HOSTNAME$ macro.

In this example the program myds receives the host name as its first argument. The following program example produces this for testing in the form of a local check. Via $1 it takes the first argument and saves it for use as an overview in the $HOST_NAME variable. This will then be inserted into the local check’s plug-in output:

~/local/bin/myds
#!/bin/sh
HOST_NAME="$1"

echo '<<<local>>>'
echo "0 Hostname - My name is ${HOST_NAME}"
Copy file content to clipboard
Successfully copied file content to clipboard!
Write access to clipboard has been denied!

The service discovery will then find a new service of the local type, in the output from which the host name will be seen:

The service discovery finds the new service, which now outputs the passed host name as information.

From here it is only a small step to a real data source program that, for example, retrieves data over HTTP using the curl command. The following placeholders are permitted in a data source program’s command line:

$HOSTNAME$

The host name as configured in the Setup.

$HOSTADDRESS$

The IP address of the host over which it will be monitored.

$_HOSTTAGS$

The list of all host tags, separated by blank characters – enclose this argument in quotes to prevent it being split by the shell.

If you have a dual-monitoring using IPv4 and IPv6, the following macros may be interesting for you:

$_HOSTADDRESS_4$

The host’s IPv4-address

$_HOSTADDRESS_6$

The host’s IPv6-address

$_HOSTADDRESS_FAMILY$

The numeral 4 if the IPv4-address is used for the monitoring, otherwise 6.

2.4. Error handling

Regardless of your actual occupation in IT—​much of your time will be spent dealing with errors and problems. Data source programs are not spared these. Especially for programs that provide data over networks it is unrealistic to expect them to be error-free.

In order that Checkmk can communicate an error to your program in an orderly way, the following apply:

  1. Any exit code other than 0 will be treated as an error.

  2. Error messages are expected on the standard error channel (stderr).

If a data source program fails,

  • Checkmk discards the output’s complete user data,

  • Checkmk sets the Check_MK service to CRIT and identifies the data from stderr as an error,

  • and the actual services remain in their old state (and will become stale over time).

We can modify the above example so that it simulates an error. With the redirection >&2 the text will be diverted to stderr, and exit 1 sets the program’s exit code to 1:

~/local/bin/myds
#!/bin/sh
HOST_NAME=$1

echo "<<<local>>>"
echo "0 Hostname - My name is $HOST_NAME"

echo "This didn't work out" >&2
exit 1
Copy file content to clipboard
Successfully copied file content to clipboard!
Write access to clipboard has been denied!

As a Check_MK service it will look like this:

If a script returns exit codes different from 0, the 'Check_MK' service will immediately CRIT (red).

Should you be writing your program as a shell script, right at the beginning you can code the set -e option:

~/local/bin/myds
#!/bin/sh
set -e
Copy file content to clipboard
Successfully copied file content to clipboard!
Write access to clipboard has been denied!

As soon as an instruction produces an error (i.e., exit code not 0), the shell immediately stops and issues the exit code 1. You have thus a generic error handling and must not check every single instruction for success.

3. Files and directories

Path Function

~/local/bin/

The repository for own programs and scripts that should be in a search path, and which can be directly executed without specifying the path. If a program is in ~/bin/ as well as in ~/local/bin/, the latter has priority.


Last modified: Tue, 18 Aug 2026 07:13:02 GMT via commit c4ea9fc95
On this page