For a monitoring system to receive more information from an endpoint other than simply that it is accessible, help is required from the target system. For example — how else can Checkmk know how full a server’s storage volume is without that system somehow providing the information? The component that provides this information is always an active piece of software — namely a monitoring agent.
There are situations where one does not need to install an agent for monitoring — since one that can be used is already present. The best example here is SNMP. All manageable network devices and appliances have a built-in SNMP agent.
As always, there is an exception: the monitoring of network services. For example — by its nature a web application can be accessed via HTTP and be monitored over the same connection. But even in this case there is usually an additional agent in use which provides supplementary server data to the monitoring.
The following table shows the four different ways that Checkmk can access services to be monitored:
Checkmk accesses the target device’s existing SNMP agent. With active queries (GET) it collects details on the system’s condition.
Checkmk has its own agents for servers and workstations. Various operating systems are supported — from the commonplace like Windows and Linux to exotics such as OpenVMS. The agents are passive and connect to TCP port 6556. Only on receiving a Checkmk server query will they be activated and respond with the required data.
Installing the Checkmk agent
Some systems allow neither an agent installation, nor do they support SNMP in a usable form. Instead of these they offer management APIs based on TELNET, SSH or HTTP/XML. Checkmk queries these interfaces via 'special agents' that run on the Checkmk-Server
Create an API account for Checkmk
Network-based services such as HTTP, SMTP or IMAP can by their very nature be queried via the network. For this Checkmk sometimes uses its own, sometimes existing plug-ins originally developed for Nagios. These are also called 'active checks'. For example,
1.1. Monitoring by events
Until now we have only discussed active monitoring — Checkmk’s showpiece discipline. There is also the reverse method: namely that by which the target system itself sends messages to the monitoring, e.g., via syslog or SNMP traps. For these functions Checkmk has the Event Console which is described in its own article.
2. The Checkmk agent
As described, you require a Checkmk agent in order to monitor servers and workstations. Agents for eleven different operating system families are currently maintained in the Checkmk project. All of these agents are components in Checkmk, and are available for downloading via the Checkmk server’s web interface. Additionally, in the Enterprise Editions there is the Agent Bakery. This enables you to ‘bake’ your own agents containing a personalised configuration and collection of plug-ins. This can even be done individually and differently for each host as required.
3. Installation via the download page
These agents are accessed via via Setup > Agents. In the Raw Edition, the menu items Linux, Windows and Other operating systems will take you directly to the download pages where you will find the pre-configured agents and agent plug-ins, in the following example to the download page for Linux, Solaris, AIX:
In the Enterprise Editions, the menu item Windows, Linux, Solaris, AIX takes you to a page that also gives you access to the Agent Bakery. In addition, however, a generic agent is always offered, which you can download immediately. Via the Related menu you can access the download pages mentioned above for the Raw Edition.
The packaged agents for Linux (in RPM and DEB file format) and for Windows (in MSI file format) are found right in the first section of the corresponding download page. After installing these packages the agent is basically ready to run, and you can begin monitoring. All further configuration is achieved via configurations data, and plug-ins are installed by downloading them from the same page and copying them into the appropriate folder.
4. Installation via the Agent Bakery
While it is true that the Checkmk agent can function ‘naked’ immediately — without needing configuration, and without plug-ins — nonetheless in some cases the agent does need to be set up. Some examples:
Restriction of access to specific IP addresses
Monitoring of Oracle data bases (a plug-in and configuration are required)
Monitoring of text log files (a plug-in, data names and a text-pattern are required)
Utilization of the hardware/software inventory (a plug-in required)
If you have one of the Checkmk Enterprise Editions you can package personalised agents with the Agent Bakery. In this way, alongside the existing agents, you can also create agent packages that contain configurations and extra plug-ins. You can install these packages with a single command. These packages are ideal for automatic distribution and installation. You can even create personalized agents for specific groups of hosts. This allows great flexibility through the use of the automatic agent updates.
You can reach the Agent Bakery via Setup > Agents > Windows, Linux, Solaris, AIX:
If you have not yet made settings for specific hosts, there is only a single default agent configuration.
Checkmk supports the Windows, Linux, Solaris and AIX operating systems.
For Linux you have a choice between the package formats RPM (for Red Hat, CentOS, SLES), and DEB (for Debian, Ubuntu), as well as a so-called 'tarball' that is simply unpacked as
Likewise, a tarball is available for AIX, however this does not include automatic integration into the
The integration must be performed manually as a one-off action.
For Solaris there is again the tarball and a PKG package.
Every agent configuration has an explicit ID: its hash. A hash’s first eight characters are displayed in the GUI. This hash will be a part of the package version and embedded in its file name. Whenever you change something in a package’s configuration or update Checkmk, the package’s hash will also be changed. In this way the operating system’s package manager recognizes that it is an update. Checkmk’s version number would not suffice in such a case.
4.2. Configuration via rules
The agent’s configuration can be altered — as is so often the case in Checkmk — via rules. These offer you the possibility of equipping different hosts with differing settings or plug-ins.
The Agent rules button takes you to a page which lists all of the rule sets that affect the agents:
Let’s take the following example: you wish to limit the list of IP addresses that are permitted to access the agent. For this you select the Generic Options > Allowed agent access via IP address (Linux, Windows) rule set. Enter one or more IP adresses as the rule’s value:
After saving, go back to the Windows, Linux, Solaris, AIX page. The button ensures that the agent will be freshly-baked. The result — you now have two individual configurations:
In the Hosts column you will find a list of hosts associated with the relevant configuration. For space reasons this list may not be complete. The VANILLA and GENERIC names have a special role. These two pseudo-hosts are always present and have the following functions:
A virtual host whose agent contains only the default configuration, to which therefore none of the agent rules apply.
A virtual host to which all rules with no defined additional conditions apply. This entry is especially useful for installing agents on hosts that have not yet been incorporated into the monitoring.
The more host-specific rules you deploy, the more different versions of agents will be built. The Agent Bakery makes sure that only such combinations of configurations are built that will be used by at least one of the available hosts.
By the way, you can also reach a host’s agent packages conveniently via the host’s properties by clicking on the host in Setup > Hosts > Hosts and selecting the entry Monitoring Agent in the Hosts menu:
Why are packages for all operating systems offered for every host? The answer is very simple: if no agent is installed on a system Checkmk naturally cannot recognise the operating system! In any case, once automatic agent updates are activated you don’t need to do anything more.
Many rules are concerned with the installation of various plug-ins. These extend the agents for the monitoring of quite specific components. Most of these are special applications such as databases, for example. Alongside the rule that activates a plug-in you will also find the settings for configuring the plug-in. Here, for example, is the rule for monitoring MySQL:
4.4. Customising agents manually
Please note that on the target system you do not manually modify the configuration files of an agent that was created by the Agent Bakery. This will work at first, but the next update of the agent will cause the the changes to be lost. However it is possible to install additional plug-ins and configuration files without problems.
5. When should an agent be updated?
Regardless of whether you monitor only a handful — or even thousands of hosts — management of the Checkmk agents on all hosts is always a larger operation. The automatic agent update in the Checkmk Enterprise Editions is however a big help. Nonetheless, you should really only update the agents when:
the update solves a problem affecting you, or
the update includes required new functions.
In order for this to be possible a general rule applies in Checkmk: newer versions of Checkmk can fundamentally handle the output of older agents.
Important: the reverse is not necessarily true. If an agent’s Checkmk version is newer than that of the monitoring server it is possible that the check plug-ins there cannot interpret the agent’s output correctly. In such a case the affected services go into an UNKNOWN status:
Even if the output suggests otherwise, please do not send a crash report in such a case.
6. Error diagnosis
6.1. Testing agents via the command line
Although the agents for the various operating systems were independently developed, from Checkmk’s point of view they all behave in the same way by opening the TCP port 6556 for queries from the monitoring server. The query protocol is absolutely simple: the server connects to the port and the data flows in a readable text format from the agent. As soon as the data transfer is completed the agent disconnects itself from the port. The agent basically reads no data from the network!
A correctly-installed agent can be very easily queried from the command line.
The best way to do this is directly from the Checkmk site that is also actively monitoring the agent.
In this way you can be certain that the server’s IP address will be accepted by agents. A suitable command is e.g.
OMD[mysite]:~$ telnet 10.1.1.2 6556 Trying 10.1.1.2... Connected to 10.1.1.2. Escape character is '^]'. <<<check_mk>>> Version: 2.0.0p5 AgentOS: linux Hostname: mycmkserver AgentDirectory: /etc/check_mk DataDirectory: /var/lib/check_mk_agent SpoolDirectory: /var/lib/check_mk_agent/spool PluginsDirectory: /usr/lib/check_mk_agent/plugins
netcat the data is returned ‘naked’.
This is useful for example, if you wish to use a script to process the data:
OMD[mysite]:~$ nc 10.1.1.2 6556 <<<check_mk>>> Version: 2.0.0p5 AgentOS: linux Hostname: mycmkserver AgentDirectory: /etc/check_mk DataDirectory: /var/lib/check_mk_agent SpoolDirectory: /var/lib/check_mk_agent/spool PluginsDirectory: /usr/lib/check_mk_agent/plugins
The output always begins with the line
Lines included in
>>> are called section headers.
These divide the agent output into sections.
Each section contains related information and is usually simply the output from a diagnosis command.
check_mk section plays a special role.
It contains general information about the agent such as e.g., its version number.
If the host is already being monitored you can also fetch the data with the
cmk -d command.
This uses the IP address configured in the Setup, allows for a possibly reconfigured port number, and also for any special agent that may be present:
OMD[mysite]:~$ cmk -d mycmkserver <<<check_mk>>> Version: 2.0.0p5
If monitoring is already running regularly for the host in question a current copy of the output can always be found in the
OMD[mysite]:~$ cat tmp/check_mk/cache/mycmkserver <<<check_mk>>> Version: 2.0.0p5
6.2. Testing agents via the web interface
You can also conduct a diagnosis of the agents via the web interface. This takes all settings into consideration and also supports SNMP devices and those queried using special agents. In effect, Checkmk simply attempts to always query via TCP port 6556 and SNMP simultaneously. You can access the connection test via the properties of the host: On the Properties of host page, select Hosts > Connection tests from the menu, and start the test by clicking Run tests:
You can try out quite a few of the settings (e.g., the SNMP community) right away, and save them when successful.