1. Introduction
With over 2000 ready-made check plug-ins and multiple methods for the monitoring of files and folder contents, the evaluation of log messages and the monitoring of recurring tasks, Checkmk is the ideal out-of-the-box solution for a wealth of monitoring tasks. Should a plug-in for a very specialized use not be available, the Checkmk community will be happy to help with custom developments provided via the Checkmk Exchange.
Nevertheless, there are always situations where a piece of hardware is too new, a piece of software is too exotic, or an organization’s own development is too individual for anyone to have already recognized the need for its integration into Checkmk. If you have reached this point, it is time to start programming your own extensions.
This article provides an overview of the options available.
These options are manifold: In some cases, for example, it is sufficient to extend a backup script by a few lines in order to output success or failure in a form that can be easily displayed in Checkmk — this means that the 'in-house development' can sometimes be completed in just a few minutes. In other cases you will need to visualize load situations with extensive graphs — in such a situation it is worth investing a few hours more.
2. Extension possibilities using own programs
The following sections show which procedures for integrating your own extensions into Checkmk are possible, and where the data collection and evaluation takes place in each case.
2.1. Local checks
Probably the simplest way to extend Checkmk is through the use of local checks. A program executed by the agent script of the monitored host prints name, state and other required information in one line. For local checks, Checkmk supports automatic service discovery. Programming is possible in any programming language without having to learn an API.
Execution: Entirely on the monitored host. You must ensure that the appropriate interpreter is available on all hosts receiving a local check, when applicable.
Thresholds: A pair of lower and upper thresholds (for transitions to WARN and CRIT respectively) can be managed by the Checkmk site.
Metrics: Multiple metrics per service are possible. Units cannot be explicitly managed, these are automatically assigned or omitted.
2.2. Native agent-based check plug-ins
The agent-based check plug-ins evaluate data supplied by the Checkmk agent. An agent plug-in collects raw data and pre-filters it, but does not perform an analysis on the collected data. This data collection can be performed in any programming language. Very common is the output as a JSON file or in the CSV format. However, you will also see many agent plug-ins that only call raw Linux system commands.
The evaluation then takes place on the Checkmk server using a check plug-in written in Python, which makes use of APIs from Checkmk. A status can thus be determined very flexibly. It is possible to use lower and upper threshold values. In addition, multiple services can be created and the status of a service can be verified by multiple checks. It is also possible to determine trends and include older values. Native check plug-ins support the automatic creation of labels and the HW/SW inventory.
Execution: Agent plug-in for data collection in any programming language on the monitored host, further evaluation by a check plug-in on the Checkmk server using the Check API.
Thresholds: Any combination of threshold values for each service.
Metrics: Any number of metrics per service with units.
2.3. Special agents
An extension of the agent-based check plug-ins are special agents. Here, no agent plug-in collects the raw data, but a program running on the Checkmk server that retrieves data from another source and converts it into Checkmk’s agent format. Special agents are used, for example, when a device to be monitored provides data relevant for monitoring as JSON or XML via a REST API. For examples of the use of special agents provided with Checkmk, see AWS, Azure, or VMware.
When programming, you access two APIs: For the configuration of ports or similar, Checkmk provides an API that allows you to specify such settings in the setup. For the data query itself, use the REST API at the external source. The evaluation on the Checkmk server is carried out as described in the previous section on native check plug-ins.
Execution: Program/script for data collection and further evaluation on the Checkmk server.
Thresholds: Any combination of threshold values for each service.
Metrics: Any number of metrics per service with units.
2.4. Native SNMP-based check plug-ins
A variant of the agent-based check plug-ins are the check plug-ins for SNMP. The difference here is that no agent section is requested and evaluated, instead certain SNMP OIDs are explicitly requested by the SNMP agent.
Execution: Data collection and further evaluation by a check plug-in on the Checkmk server using the Check API.
Thresholds: Any combination of threshold values for each service.
Metrics: Any number of metrics per service with units.
As the SNMP protocol is inherently very inefficient, we recommend only using SNMP if no other access to the monitoring data is possible. For example, if a device also provides the same data via a REST API, you should build a special agent for this.
2.5. Legacy Nagios check plug-ins
Nagios check plug-ins can be found in two places in Checkmk: As active checks, to check the accessibility of certain services from the Checkmk server, and as an MRPE extension of the Windows or Linux agents for checking such services from a host — if the host/services are not accessible from the Checkmk server.
Programming is possible in any language.
Execution: Completely on the monitored host (via MRPE) or completely on the Checkmk server (active check).
Thresholds: Threshold values only when used as active check.
Metrics: Metrics only when used as active check.
Due to a number of disadvantages such as cumbersome troubleshooting, we only recommend re-implementation if a full compatibility with Nagios is required. In all other cases, use native check plug-ins or — for simple queries — use local checks. Detailed documentation of the output formats can be found at Monitoring-Plugins.org.
3. Additional articles
3.1. The spool directory
Checkmk provides yet another mechanism for generating agent data: Have a program directly write a text file in the Checkmk agent format. Stored in the spool directory, the Checkmk agent transfers the contents of this file with the rest of the agent output.
With the spool directory you can, for example, have backup scripts write status and statistics for a local check or a check plug-in directly on completion. This saves detours via the evaluation of log files.
When developing your own check plug-ins, spool files help to simulate particular outputs from your agent plug-in.
3.2. The piggyback mechanism
The piggyback mechanism is used when one host has information about another. A specially-formatted agent section is then assigned to the relevant host when evaluating the agent output.
For virtual machines, the piggyback mechanism is used to merge data collected by the virtualization software with the data from the monitoring within the virtual machine.
3.3. Checkmk extension packages (MKPs)
If you have programmed your own extensions and want to version them and then forward them, you have the option of bundling an extension with its associated files in Checkmk extension packages (MKPs). You must also use this package format if you want to offer these extensions in the Checkmk Exchange.
3.4. The Bakery API
In many cases, you will want to provide agent plug-ins with additional configuration. Or you may want to execute specific installation scriptlets depending on the settings made in the configuration of Checkmk.
If you use the Agent Bakery for the distribution of agent packages, the Bakery API provides you with a programming interface with which settings made in Checkmk can easily transferred to other hosts being monitored.
4. Contributing to Checkmk
If you program your own extensions, we recommend that you first submit them to the Checkmk Exchange. Here you remain the owner and contact person and you can easily provide new versions. As the coding quality requirements for the Exchange are not as high as for check plug-ins delivered with Checkmk, you can easily try out new ideas with a wide audience via the Exchange.
If at some point you think that your check plug-in should become an integral part of Checkmk, first read the Contributing to Checkmk document.