In this article, the screenshots and the GUI navigation described have not yet been updated to Checkmk version 2.0.0. However, nothing fundamental has changed in the described functions themselves and most of the functions can be found quickly with the Monitor or Setup menu search of the Checkmk 2.0.0 user interface. We will update this article as soon as possible.

1. Introduction

Checkmk normally accesses its agents over a simple TCP-connection to Port 6556. The ‘protocol’ is trivial: as soon the server opens the connection the agent sends its data and then closes the port. The transferred data is in the form of readable ASCII text. This is esentially only the transfer of text data from agents to the server.

This radically simple principle enables the craziest variations with which even the most unusual challenges can also be fulfilled – for example, those arising out of special architectures for security or networks. A variation of this, the transfer of data via SSH, will be described in the article on the Linux agents. Here are a few further examples of how one can get agent data to the Checkmk server:

  • via email

  • via HTTP-access from the server

  • via HTTP-upload from the agents

  • via access to a file that has been copied to the server using rsync

  • via a script that uses HTTP to retrieve the data from a website

The universal methods for connecting such transfer procedures to Checkmk are the Datasource programs. The idea is very simple: one gives a command line of a command to Checkmk. Instead of connecting to Port 6556, Checkmk executes this command. This produces the agent data on the Standard Output, which is then processed by Checkmk in exactly the same way as if it had come from a ‘normal’ agent.

2. Writing from datasource programs

2.1. The simplest possible program

The writing and installation of a datasource program is not difficult. Any Linux-supported script and program language can be used. The program is best stored in the local/bin directory, where it will always be found automatically without the need to specify a data path.

The following first very basic example is called myds and it generates simple, fictional monitoring data. These consist of one section <<<df>>>, which contains the information for a single file system, and which has a size of 100kB and the name My_Disk. It is coded as a shell script of three lines:

echo '<<<df>>>'
echo 'My_Disk  foobar  100 70 30  70% /my_disk'

Don’t forget to make the program executable:

OMD[mysite]:~$ chmod +x local/bin/myds

Now create a test host in WATO – e.g., myserver125. This does not require an IP-address. In order to avoid Checkmk attempting to close myserver125 via DNS, enter this name as an explicit ‘IP-address’.

Next add a rule in the Datasource Programs > Individual program call instead of agent access rule set which applies to this host, and enter myds as an executable program:

ds program

When you now go to the host’s service configuration in WATO, exactly one service ready to start monitoring should be listed:

ds program discovery

Add this service into the monitoring, activate the changes, and your first datasource program will be running. For a test, as soon as you alter the data being generated by the program the My_Disk file system’s next check will immediately show this.

2.2. Error diagnosis

If something is not functioning correctly, the host’s configuration can be checked by entering cmk -D in the command line (if your rule allows this.):

OMD[mysite]:~$ cmk -D myserver125

Addresses:              myserver125
Tags:                   /wato/, cmk-agent, ip-v4, ip-v4-only, lan, prod, site:beta, tcp, wato
Host groups:
Contact groups:         all
Type of agent:          Datasource program: myds

With a cmk -d you can trigger the retrieval of the agent data as well as the execution of your program:

OMD[mysite]:~$ cmk -d myserver125
My_Disk  foobar  100 70 30  70% /my_disk

A duplicated -v should generate a message that your program will be invoked:

OMD[mysite]:~$ cmk -vvd myserver125
Calling external program myds
My_Disk  foobar  100 70 30  70% /my_disk

2.3. Transferring a host’s name

The program in our example actually works, but is not very useful as it always produces the same data, regardless of which host it is invoked for.

A real program that, for example, retrieves data from somewhere, requires at least the name of the host from where it should retrieve the data. By coding $HOSTNAME$ as a placeholder in the command line you can allow this to be transferred:

ds program host

In this example the program myds receives the host name as its first argument. The following program example produces this for testing in the form of a ‘local check’. Via $1 it takes the first argument and saves it for use as an overview in the $HOST_NAME variable. This will then be inserted into the local check’s plug-in output:


echo "<<<local>>>"
echo "0 Hostname - My name is $HOST_NAME"

The service discovery will then find a new service of the local type, in the output from which the host name will be seen:

ds program discovery host

From here it is only a small step to a real datasource program that, for example, retrieves data over HTTP using the curl command. The following placeholders are permitted in a datasource program’s command line:


The hostname, as it is configured under WATO.


The IP-address of the host over which it will be monitored.


The list of all of the host’s attributes, separated by blank characters – enclose this argument in quotes to prevent it being split by the shell.

If you have a dual-monitoring using IPv4 and IPv6, the following macros may be interesting for you:


The host’s IPv4-address


The host’s IPv6-address


The numeral 4 if the IPv4-address is used for the monitoring, otherwise 6.

2.4. Error handling

Regardless of your actual occupation in IT – much of your time will be spent dealing with errors and problems. Datasource programs are not spared these. Especially for programs that provide data over networks it is unrealistic to expect them to be error-free.

In order that Checkmk can communicate an error to your program in an orderly way, the following apply:

  1. Any exit code other than 0 will be treated as an error.

  2. Error messages are expected on the standard error channel (stderr).

If a datasource program fails,

  • Checkmk discards the output’s complete user data,

  • Checkmk sets the Checkmk-service to CRIT and identifies the data from stderr as an error,

  • and the actual services remain in their old state (and will age with time).

We can modify the above example so that it simulates an error. With the redirection >&2 the text will be diverted to stderr, and exit 1 sets the program’s exit status to 1:


echo "<<<local>>>"
echo "0 Hostname - My name is $HOST_NAME"

echo "This didn't work out" >&2
exit 1

In Checkmk-Service it will look like this:

ds program error

Should you be writing your program as a shell script, right at the beginning you can code the set -e option:

set -e

As soon as an instruction produces an error (i.e., exit code not 0), the shell immediately stops and issues the exit code 1. You have thus a generic error handling and must not check every single instruction for success.

3. Special agents

A number of often-required datasource programs are delivered with Checkmk. These generate agent outputs not just by calling a normal Checkmk-agent in a roundabout way, rather they have been specially conceived for the querying of particular hardware or software.

Partly because these programs require quite complex parameters, we have defined special WATO-rule sets with which you can configure them directly. All of these rules can be found under Host- & Serviceparamters > Datasource programs:

ds program rules

These programs are also known as ‘Special Agents’, because they are a special alternative to the normal Checkmk-agents. As an example, let us take the monitoring of NetApp-Filers. These do not allow the installation of Checkmk-Agents. The SNMP-interface is slow, flawed and incomplete. There is however a special HTTP-interface which provides access to all monitoring data.

The agent_netapp special agent accesses via this interface and is set up as a datasource program using the Check NetApp via WebAPI rule set. It is important that in WATO the host retains the Checkmk Agent (Server) setting.

The data required by the special agent can then be entered into the rule’s content. This is almost always some sort of access data. With NetApp-agents there is also an additional check box for the recording of performance data (which here can be quite comprehensive):

ds program netapp

There are rare occasions in which it is desired that both a special agent, as well as the normal agents are to be queried. An example for this is the monitoring of VMWare ESXi over the vCenter. This latter is installed on a (usually virtual) Windows machine, on which reasonably enough a Checkmk-Agent is also running.

ds program vcenter

The special agents are installed under share/check_mk/agents/special. If you wish to modify such an agent, first copy the file with the same name to local/share/check_mk/agents/special and make your changes in that new version.

4. Files and directories



The repository for an operation’s own programs and scripts that should be in a search path, and which can be directly executed without specifying the path. If a program is in bin/ as well as in local/bin/, the latter has priority.


The special agents provided with Checkmk are installed here.


The repository for your own modified special agents.

On this page