In this article, the screenshots and the GUI navigation described are not yet updated to Checkmk version 2.0.0. However, nothing fundamental has changed in the described functions themselves and most of the functions can be found quickly with the Monitor or Setup menu search of the Checkmk 2.0.0 user interface. We will update this article as soon as possible.
Checkmk normally accesses its agents over a simple TCP-connection to Port 6556. The ‘protocol’ is trivial: as soon the server opens the connection the agent sends its data and then closes the port. The transferred data is in the form of readable ASCII text. This is esentially only the transfer of text data from agents to the server.
This radically simple principle enables the craziest variations with which even the most unusual challenges can also be fulfilled – for example, those arising out of special architectures for security or networks. A variation of this, the transfer of data via SSH, will be described in the article on the Linux agents. Here are a few further examples of how one can get agent data to the Checkmk server:
via HTTP-access from the server
via HTTP-upload from the agents
via access to a file that has been copied to the server using
via a script that uses HTTP to retrieve the data from a website
The universal methods for connecting such transfer procedures to Checkmk are the Datasource programs. The idea is very simple: one gives a command line of a command to Checkmk. Instead of connecting to Port 6556, Checkmk executes this command. This produces the agent data on the Standard Output, which is then processed by Checkmk in exactly the same way as if it had come from a ‘normal’ agent.
2. Writing from datasource programs
2.1. The simplest possible program
The writing and installation of a datasource program is not difficult.
Any Linux-supported script and program language can be used. The program is best stored
local/bin directory, where it will always be found automatically
without the need to specify a data path.
The following first very basic example is called
myds and it generates
simple, fictional monitoring data.
These consist of one section
<<<df>>>, which contains the
information for a single file system, and which has a size of 100kB and the name
It is coded as a shell script of three lines:
!/bin/sh echo '<<<df>>>' echo 'My_Disk foobar 100 70 30 70% /my_disk'
Don’t forget to make the program executable:
OMD[mysite]:~$ chmod +x local/bin/myds
Now create a test host in WATO – e.g.,
myserver125. This does not
require an IP-address. In order to avoid Checkmk attempting to close
myserver125 via DNS, enter this name as an explicit ‘IP-address’.
Next add a rule in the Datasource Programs > Individual program call instead of agent access
rule set which applies to this host, and enter
myds as an executable
When you now go to the host’s service configuration in WATO, exactly one service ready to start monitoring should be listed:
Add this service into the monitoring, activate the changes, and your first
datasource program will be running. For a test, as soon as you alter the data
being generated by the program the
My_Disk file system’s next check
will immediately show this.
2.2. Error diagnosis
If something is not functioning correctly, the host’s configuration can be checked
cmk -D in the command line (if your rule allows this.):
OMD[mysite]:~$ cmk -D myserver125 myserver125 Addresses: myserver125 Tags: /wato/, cmk-agent, ip-v4, ip-v4-only, lan, prod, site:beta, tcp, wato Host groups: Contact groups: all Type of agent: Datasource program: myds
cmk -d you can trigger the retrieval of the agent data
as well as the execution of your program:
OMD[mysite]:~$ cmk -d myserver125 <<<df>>> My_Disk foobar 100 70 30 70% /my_disk
-v should generate a message that your program will be invoked:
OMD[mysite]:~$ cmk -vvd myserver125 Calling external program myds <<<df>>> My_Disk foobar 100 70 30 70% /my_disk
2.3. Transferring a host’s name
The program in our example actually works, but is not very useful as it always produces the same data, regardless of which host it is invoked for.
A real program that, for example, retrieves data from somewhere,
requires at least the name of the host from where it should retrieve the data.
$HOSTNAME$ as a placeholder in the command line you can allow
this to be transferred:
In this example the program
myds receives the host name as its first argument.
The following program example produces this for testing in the form of a ‘local check’.
$1 it takes the first argument and saves it for use as an overview in
This will then be inserted into the local check’s plug-in output:
!/bin/sh HOST_NAME=$1 echo "<<<local>>>" echo "0 Hostname - My name is $HOST_NAME"
The service discovery will then find a new service of the
in the output from which the host name will be seen:
From here it is only a small step to a real datasource program that, for example, retrieves data over HTTP
curl command. The following placeholders
are permitted in a datasource program’s command line:
The hostname, as it is configured under WATO.
The IP-address of the host over which it will be monitored.
The list of all of the host’s attributes, separated by blank characters – enclose this argument in quotes to prevent it being split by the shell.
If you have a dual-monitoring using IPv4 and IPv6, the following macros may be interesting for you:
The host’s IPv4-address
The host’s IPv6-address
2.4. Error handling
Regardless of your actual occupation in IT – much of your time will be spent dealing with errors and problems. Datasource programs are not spared these. Especially for programs that provide data over networks it is unrealistic to expect them to be error-free.
In order that Checkmk can communicate an error to your program in an orderly way, the following apply:
Any exit code other than 0 will be treated as an error.
Error messages are expected on the standard error channel (
If a datasource program fails,
Checkmk discards the output’s complete user data,
Checkmk sets the Checkmk-service to CRIT and identifies the data from
stderras an error,
and the actual services remain in their old state (and will age with time).
We can modify the above example so that it simulates an error.
With the redirection
>&2 the text will be diverted
exit 1 sets the program’s exit status to 1:
!/bin/sh HOST_NAME=$1 echo "<<<local>>>" echo "0 Hostname - My name is $HOST_NAME" echo "This didn't work out" >&2 exit 1
In Checkmk-Service it will look like this:
Should you be writing your program as a shell script, right at the beginning
you can code the
set -e option:
!/bin/sh set -e
As soon as an instruction produces an error (i.e., exit code not 0), the shell immediately stops and issues the exit code 1. You have thus a generic error handling and must not check every single instruction for success.
3. Special agents
A number of often-required datasource programs are delivered with Checkmk. These generate agent outputs not just by calling a normal Checkmk-agent in a roundabout way, rather they have been specially conceived for the querying of particular hardware or software.
Partly because these programs require quite complex parameters, we have defined special WATO-rule sets with which you can configure them directly. All of these rules can be found under Host- & Serviceparamters > Datasource programs:
These programs are also known as ‘Special Agents’, because they are a special alternative to the normal Checkmk-agents. As an example, let us take the monitoring of NetApp-Filers. These do not allow the installation of Checkmk-Agents. The SNMP-interface is slow, flawed and incomplete. There is however a special HTTP-interface which provides access to all monitoring data.
agent_netapp special agent accesses via this interface and is
set up as a datasource program using the Check NetApp via WebAPI rule set.
It is important that in WATO the host retains the Checkmk Agent (Server) setting.
The data required by the special agent can then be entered into the rule’s content. This is almost always some sort of access data. With NetApp-agents there is also an additional check box for the recording of performance data (which here can be quite comprehensive):
There are rare occasions in which it is desired that both a special agent, as well as the normal agents are to be queried. An example for this is the monitoring of VMWare ESXi over the vCenter. This latter is installed on a (usually virtual) Windows machine, on which reasonably enough a Checkmk-Agent is also running.
The special agents are installed under
If you wish to modify such an agent, first copy the file with the same name
local/share/check_mk/agents/special and make
your changes in that new version.
4. Files and directories
The repository for an operation’s own programs and scripts that should be in a search path, and which can be directly executed without specifying the path. If a program is in
The special agents provided with Checkmk are installed here.
The repository for your own modified special agents.