Writing your own check plug-ins

What options do you have for implementing an effective monitoring here? Well, you could of course contact our support team and request that they develop a suitable plug-in for you — but naturally it is quicker if you can do it yourself.

You have four options:

This article will show you how to develop real Checkmk check plug-ins — along with everything associated with them. Here we show you how to use the newly-developed API for programming plug-ins in version 2.0.0 of Checkmk.

Do you already have experience with developing check plug-ins for Checkmk Version 1.6.0 or earlier? If so, here is a concise overview of all of the changes introduced in the new Check API available from Version 2.0.0:

Yes — the API for the development of check plug-ins valid up to version 1.6.0 of Checkmk will be supported with some minor restrictions for a few more years, because a significant number of plug-ins have been developed with it. During this time, Checkmk will offer both APIs in parallel. Details can be found in the #10601 work.

Nevertheless, we do recommend the new API for the development of new plug-ins, as it is more consistent and logical, better documented and is the most future-proof solution in the long term.

Check plug-ins evaluate the data from the Checkmk agents. And that is why, before we leap into action here, we should first look at an overview of the types of agents that Checkmk actually recognises:

If you feel like programming check plug-ins, you need to satisfy the following prerequisites:

As preparation, the following articles are recommended:

At the beginning of any check programming activity is the necessary research! This means that we have to find out how we can get the information we need for monitoring. With Linux, this will often involve command line commands. In Windows, PowerShell, VBScript or WMI can help, and with SNMP we have to find the right OIDs (there is an own chapter for this).

Unfortunately, there is no general procedure for determining the correct command, so we do not want to spend too much time on the subject here, we will however briefly explain how it works for a USB stick.

First we log in to the host we want to monitor. Under Linux, the agent runs as the root user by default — which is why we perform all our tests as root. For our task with the USB stick, there are convenient symbolic links in the directory /dev/disk/by-id. These point to all the Linux block devices. And a plugged-in USB stick is one such device. In addition, you can tell by the ID of the prefix usb- when a block device is a USB device.

The following command lists all entries in this directory:

So — and now the whole thing with the plugged in USB stick:

Actually, we would be finished with that and could transport this whole output via the Checkmk agent to the Checkmk server and have it analysed there as well, because in Checkmk the following recommendation always applies: always let the server do the complex work, so keep the agent plug-in as simple as possible.

But there is still too much hot air in here. Transferring unnecessary data is always undesirable. Avoiding unnecessary transfers saves network traffic, memory, computing time and also makes everything clearer. That is simply a better way of doing things!

First, we can omit the -l. This already makes the output of ls much leaner:

Now again, the multi-column structure is disturbing, but this is only because the ls command recognises that it is running in an interactive terminal. Later, as part of the agent, it will output the data in a single column. But we can also easily force this here with the -1 option (for output in one column):

If you look closely, you will see not only the block devices themselves, but also any partitions that exist there. These are the entries that end in -part1, -part2, etc. We do not need these for our check and can get rid of them quite easily with a grep. There we take the -v option for negative logic:

Here you can now see much more clearly that in our example there are in fact exactly three devices when the USB stick is plugged in.

Perfect! We now have a clear list of all block devices which has been compiled with a simple command. That’s all we need.

We have again omitted the -1 in the last command, because ls now writes into a pipe and outputs a single column by itself. And grep needs the -- because otherwise it would interpret the word -part as the four options -p, -a, -r and -t.

And by the way: Why don’t we simply 'grep' for usb in addition so that only USB devices are output? Well, of course we could do that. But for one thing, our example then becomes increasingly boring, and besides, it is somehow more reassuring to get some content in the section in a normal situation and not simply nothing. In this way one can see immediately on the Checkmk server that the agent plug-in is working correctly.

In order for us to be able to retrieve this data from the Checkmk server, we need to make the new command part of the Checkmk agent on the system being monitored. We could of course simply edit the /usr/bin/check_mk_agent file there and include that. However, this would have the disadvantage that our command would disappear again when we update the agent’s software because the file will be replaced at that point.

It is therefore better if we make an agent plug-in. This is even simpler. All we need is an executable file with our command in the /usr/lib/check_mk_agent/plugins directory.

And one more point is important: we can’t just output our data like this. What we still need is a section header. This is a specially-formatted line that contains our new check’s name. By means of these section headers, Checkmk can later recognise where this plug-in’s data begins and the previous plug-in’s data ends.

So now we need a meaningful name for our new check. This name is limited to lower case letters (only a-z, no accents, no umlauts), underscores and numbers and must be unique. Avoid name collisions with existing check plug-ins. If you are curious about which names already exist, in a Checkmk site you can list them on the command line with cmk -L:

The second column shows how the respective check plug-in obtains its data.

For our example, let’s choose the name linux_usbstick. In this example the section header must look like this:

We can simply output this with echo. If we then don’t forget the 'Shebang' (this is not a venomous sting from the desert planet but an abbreviation for sharp and bang — the latter being an abbreviation for the exclamation mark!), by which Linux recognises that it should execute the script with the shell, in which case our plug-in will look like this:

We have now simply used the file name linux_usbstick, even though it doesn’t really matter. But one thing is still very important: Make the file executable!

Of course, you can easily try out the plug-in by manually by entering the complete path as a command:

As always, the next most important tasks are testing and troubleshooting. It is best to proceed in three steps:

Testing the agent locally is very simple — as root user call the command check_mk_agent. The new section should appear somewhere in the output from this:

Here is an excerpt from that output which contains the new section:

By appending less you can scroll through the output (press the space bar to scroll, / to search and Q to exit):

The third test is then performed directly from the Checkmk site. Include the host in the monitoring (e.g. as myserver01) and then retrieve the agent data with cmk -d. You should get the same output here:

By the way: grep has the -A option to output a few more lines after each hit. This allows you to conveniently search and output the section:

If this works, your agent is now ready! And what have we done to achieve this? We simply created a three-line script with the path /usr/lib/check_mk_agent/plugins/linux_usbstick and made it executable!

Everything that follows now only takes place on the Checkmk server: There we will write the actual check plug-in.

Preparing the agent is the most complicated part, but it is only half the battle. Now we have to teach Checkmk how to handle the information and the new agent section, which services it should generate, when they should go to OK or CRIT, etc. We do all this by programming a check plug-in in Python.

For your own check plug-ins you will find a directory prepared in the local hierarchy of the site directory. This is local/lib/check_mk/base/plugins/agent_based/. Here in the path, base means the part of Checkmk that is responsible for actually monitoring and alerting. The agent_based is for all plug-ins that relate to the Checkmk agent (so not alerting plug-ins, for example). The easiest way to work with this is to switch to it:

This directory belongs to the site user and is therefore editable by you. You can edit your plug-in with any text editor installed on the Linux system.

So let’s create our plug-in here. The convention is that the file name reflects the agent section’s name. Mandatory is that the file ends with .py, since from Checkmk version 2.0.0 onwards the plug-ins will always be real Python modules.

First, we need to import the functions needed for the plug-ins from other Python modules. The simplest method for this is with a *. As you might guess, there is also a version number of the API for plug-in programming here. This will be version 1 until further notice, and is abbreviated here to v1:

This versioning allows us to eventually provide future new versions of the API parallel to the previous ones, so that existing check plug-ins continue to work without problems.

In the simplest case, you skip explicitly declaring the section. If you want to implement a parse function (which professional developers would always advise you to do), see the section on parse functions for more information.

In order for Checkmk to know that the new check exists, it must be registered. This is done by calling the function register.check_plugin. In doing so, you must always specify at least four things:

So for our check the registration will look like this:

It’s best not to try this out just yet, because of course we still have to write the discover_linux_usbstick and check_linux_usbstick functions beforehand, and these functions must appear in the source code before the above declaration.

A special feature of Checkmk is the automatic discovery of services to be monitored. In order for this to work, each check plug-in must define a function that detects whether a service of this type or which services of this type are to be created for the host in question on the basis of the agent’s output.

The discovery function is always called when the service discovery is carried out for a host. This function then decides whether or which services are to be created. In the standard procedure, it receives exactly one argument with the name section. This contains the data of the agent section in a parsed format (more on this later).

We implement the following simple logic: If the agent section linux_usbstick exists, then we also create a matching service. This service will then automatically appear on all hosts where our agent plug-in has been rolled out. We recognise the presence of the section simply by the fact that our discovery has actually been invoked!

The discovery function must return an object of the type Service for each service to be created using yield (not with return). For checks that can only occur once per host, no further information is needed:

So now we can come to the actual check function, which finally decides on the basis of current agent outputs which state a service should assume. Since our check has no parameters and there is only ever one per host, our function is also called with the single argument section.

Since we really need the content this time, we have to deal with the format of this argument. Unless you have explicitly defined a parse function Checkmk will parse each line of the section into a list of words using spaces. The whole thing then in turn becomes a list of these word lists. So the end result is that we will always have a list of lists.

In the simple case in which our agent plug-in only finds two devices, it will then look like this (here there is only one word per line):

The check function now goes through line by line and looks for a line whose first (and only) word begins with usb-SCSI_DISK. If this is found, the state will become CRIT. Here is the implementation:

And here is the explanation:

And here is the complete plug-in one more time:

And this is the plug-in for the Linux agent:

In our example, we have built a very simple check that creates a service on a host — or not. A very common situation is, of course, that there can be several services with one check on one host.

The most common example of this is the file systems for a host. The plug-in named df creates one service per file system on the host. To distinguish these services, the mount point of the file system (e.g. /var), or the drive letter (e.g. C:) is built into the service name. This then results in the service name being, e.g. filesystem /var, or filesystem C:. The word /var or C: is referred to here as the item. So we also speak of a check with items.

If you want to build a check with multiple items, you need to implement the following things:

To be able to test the whole thing practically, we will simply build another agent section that only outputs game data. A small shell script is sufficient for this. The section should be called foobar in this example:

From foobar, there are three sectors to be found here: West, East and North (whatever that means). In each sector there are a number of seats, some of which are occupied (e.g. in West 100 of 100 seats are occupied).

Now we will create a matching check plug-in for this. The registration is as usual, but with the important difference that the service name now contains exactly one %s. At this position the item’s name will be inserted later by Checkmk:

The discovery function now has the task of determining the items to be monitored. As usual, it receives the section argument. And again, this is a list of lines, which in turn are lists of words.

In our example the list looks like this:

You can loop through such a list with Python and give meaningful names to these three words grouped in each line:

In each line, the first word — here the sector — is our item. Whenever we find an item, we return that with yield, creating an object of type Service that gets the sector name as its item. The underscore indicates that for now we don’t care about the other two columns in the output, since in a discovery it ultimately doesn’t matter how many slots are occupied.

Overall it looks like this:

Of course, it would be easy to omit some lines here on the basis of arbitrary criteria. Maybe there are sectors which have a size of 0 and which you would never want to monitor? Simply omit such rows so that no item will be generated for them.

Then later, when the host is being monitored, the check function is called separately for each service — and thus for each item. Therefore, in addition to the section, it also receives the item argument with the item it is looking for. Now we go through all of the lines one after the other. When doing so, we will find the line that corresponds to the desired item:

Now all that is missing is the actual logic which determines when the item should in fact be OK, WARN or CRIT. We do that here like this:

The occupied and total slots always appear as the second and third words in each line. However, here we are dealing with strings, not numbers — but we need numbers to be able to compare and calculate. We therefore convert the strings into numbers using int().

We then return the check result by supplying an object of type result via yield. This takes the parameters state and summary:

In this context, please note the following:

By the way, for the common situation where you want to check a simple metric for thresholds, there is a helper function called check_levels. This helper function is explained in the Check API documentation, which you can access in Checkmk via Help > Plugin API reference > Agent based API ("Check API").

Now let’s first try out the discovery. For the sake of clarity we will restrict this action to our plug-in by using the --detect-plugins=foobar option:

And now right away we can test the checking process (here also limited to foobar):

And here again our example in full. To avoid errors due to undefined function names, the functions must always be defined before registering.

Not always, but very often checks work with numbers. With its graphing system Checkmk has a component to store, evaluate and display such numbers. This works completely independently from the generation of any resulting OK, WARN or CRIT states.

Such measured values — or metrics — are determined by the check function and simply returned as an additional result. For this purpose the Metric object is used, which requires at least the two arguments name and value. Here is an example:

Furthermore there are two optional arguments. With the argument levels you can provide information about thresholds for WARN and CRIT, in the form of a pair of two numbers. This is then usually plotted on the graph as a yellow and a red line. The first number (yellow line) represents the warning threshold, the second (red line) the critical one. The convention is that the check already goes to WARN when the warning threshold is reached (analogous for CRIT).

The coding could then look like this (here with hard coded thresholds):

Notes:

Analogous to the threshold values, you can also provide the graphing system with information about a range of possible values. This denotes the smallest and largest possible value. This is done in the boundaries argument, where None can also be optionally used here for one of the two boundaries.

Example:

And now once again our check function from the above example, but this time with the return of metric information including threshold values and a value range (this time of course not with fixed but with calculated values):

The correct handling of errors (unfortunately) consumes a large chunk of the programming work. The good news is that the Checkmk API already does most of the work for you. Consequently, in most cases, it is important for you to simply not deal with errors.

When Python gets into a situation that is in some way unexpected , it responds with what is known as an exception. Here are a few examples:

Here the important general rule applies: Don’t capture exceptions yourself! Checkmk will always do this for you in a consistent and efficient way — in most cases accompanied by a crash report. Such a report will look like this, for example:

By clicking on the icon, the user is navigated to a page where they can

Submitting the report to Checkmk GmbH of course only makes sense for check plug-ins which are official Checkmk components. But you can also ask your users to simply send you the data. The users can then help you to find the error. It is often the case that a check plug-in works for you, but other users may experience sporadic errors. Working together you can then usually identify these problems very easily.

But if you were to intercept the exception yourself, all of this information would simply be unavailable. You would perhaps set the service to UNKNOWN and issue an error message, but all of the background circumstances that led to an error (e.g. the data from the agent) would simply be invisible.

If you run your plug-in on the command line, no crash reports will be generated — you will only see the summarized error message:

BUT: if you simply append the --debug option to this, you will then receive the Python stack trace:

The question is how to react when the output from the agent is not in the form you would normally expect — whether it is from the 'real' agent or when the data comes via SNMP. Let’s assume that you always expect three words per line. What should you do if only two words were to arrive?

Now — if this is a permitted and familiar agent behaviour, then of course you need to capture that and employ case discrimination.

If, however, this is not actually allowed …​ then it is best to treat the line as if it always contains three words, e.g. with:

If there should ever be a line that does not consist of exactly three words, a nice exception will be generated and you will receive the very helpful crash report that was just mentioned.

What if the agent outputs data correctly, but the item to be checked is missing? So, like this, for example:

If the item you are looking for is not there, the loop is run and Python just falls out of the back at the end of the function without 'yielding' a result. And that’s exactly the correct procedure! Because Checkmk recognizes that the item to be monitored is missing and with UNKNOWN generates the correct status and a suitable standard text for it.

Developing checks that work with SNMP is very similar to agent-based checks, except that you still need to specify which SNMP ranges (OIDs) the check requires. If you don’t as yet have any experience with SNMP, we strongly recommend reading the article on Monitoring via SNMP as preparation at this point.

The process of discovery and checking via SNMP is somewhat different from that for a normal agent. Because unlike there — where the agent sends all of the relevant information on its own — with SNMP we have to say exactly which data ranges we require. A complete dump of all data would be theoretically possible (via SNMP walk), but this process can take minutes for fast devices and over an hour for complex switches. Thus, this is not viable for checking or even for discovery. Checkmk therefore proceeds in a more targeted manner.

Before we continue, we want to say a word about the infamous SNMP MIBs, because there are many prejudices about these. Right at the beginning, some good news: Checkmk doesn’t need them. Really! But they are an important aid in being able to develop an SNMP check.

So what is a MIB? Literally, the abbreviation means Management Information Base — somewhat meaningless really. To be concrete, a MIB is a quite easy to read text file which describes a certain subtree in the SNMP world. Namely, it states which branch in the tree — that is, which OID — has which meaning. This includes a name for the OID, a comment on what values it can take (e.g. for enumerated data types, where things like 1=up, 2=down, etc. are defined) and sometimes a useful comment.

Checkmk provides a set of freely-available MIB files. These describe very general areas in the global OID tree, but do not contain any vendor-specific areas. Therefore they are of not much help for self-developed checks.

So try to find the MIB files relevant for your particular device somewhere on the manufacturer’s web pages or even on the device’s management interface, and install these in the Checkmk site in local/share/check_mk/mibs. You can then have SNMP walks convert OID numbers to names and can thus more quickly find where the data of interest for your monitoring is located. Also, MIBs, if done carefully, contain interesting information in their comments, as we noted above. You can easily read an MIB file with a text editor or with less.

The crucial prerequisite for developing a plug-in is, of course, that you know which OIDs contain the necessary information. The first step in doing this (if the device doesn’t refuse) is to perform a complete SNMP walk. This will retrieve all of the available data via SNMP.

Checkmk can accomplish this very easily for you. To do so, first include the device (or one of the devices) for which you want to develop a plug-in into your monitoring. Let’s say this device is called mydevice01. Check in the device’s basic functions to make sure that it can be monitored. As a minimum, the SNMP Info and Uptime services need to be found, and probably at least one Interface as well. This is how you make sure that SNMP access works cleanly.

Then switch to the command line in the Checkmk site. Here you can perform a complete walk with the following command. We recommend using the -v (verbose) option when doing this for the very first time:

As mentioned earlier, such a complete walk can take minutes or even hours — although the latter is rare. So don’t become nervous if it takes a while to complete this process. The walk will be saved in the file var/check_mk/snmpwalks/mydevice01. This will be a easily-readable text file that starts something like this:

In each line there is an OID and then its value. And right there in the first line you find the most important one, namely the sysDescr.

Now the OIDs themselves are not very informative. If the correct MIBs are installed, you can have them converted to names in a second step with the cmk --snmptranslate command. It is best to redirect the result — which would otherwise appear in the terminal — to a file:

The translated file reads like the original walk, but has a translated value for the OID on each line after the -->:

Example: the OID .1.3.6.1.2.1.1.4.0 has the translated name SNMPv2-MIB::sysContact.0. This is an important hint — the rest is then practice, experience and of course experimentation.

So, once you have determined the necessary OIDs, it’s on to the actual development of the plug-in. This is done in three steps:

The first two steps are performed by registering an SNMP section. You do this by calling register.snmp_section(). Here you specify at least three arguments: the name of the section (name), the details for the SNMP detect detect, and the OID branches needed for actually monitoring (fetch). Here is an example of a hypothetical check plug-in with the name foo:

The most important part of the SNMP declaration is the specification of which OIDs are to be fetched for the monitoring. In almost all cases, a plug-in only needs selected branches from a single table to do this. Let’s consider the following example:

The keyword base specifies an OID prefix here. All necessary data is below. At oids you then specify a list of sub-OIDs to be fetched from there. In the above example, a total of three SNMP walks are then made, namely on starting from the OIDs .1.3.6.1.4.1.35424.1.2.4.0 and .1.3.6.1.4.1.35424.1.2.5.0 and .1.3.6.1.4.1.35424.1.2.8.0. It is important that these walks fetch the same number of variables and that they also correspond to each other. This means that, for example, the nth element from each of the walks corresponds to the same monitored object.

Here is an example from the check plug-in snmp_quantum_storage_info:

Here, the vendor ID, the product ID, the product revision and the serial number are retrieved from each storage device.

The discovery and check function is presented with this data as a table, i.e. as a list of lists. The table is mirrored so that you have all of the data for each item per entry in the outer list. Each entry has as many items as you specified in oids. This allows you to loop through the list in a very practical way, e.g.:

Please note:

We will describe here in the future:

In the summary or details for a service, numbers are often output. To make it as easy as possible for you to format them nicely and correctly, and also to standardize the output from all check plug-ins, there are helper functions for rendering different kinds of sizes. All of these are sub-functions of the render module and are consequently called with render.. For example, render.bytes(2000) results in the text 1.95 KiB.

What all of these functions have in common is that they get their values in a so-called canonical or natural unit. Thus one must never think, and there are no difficulties or errors with the conversion. For example, times are always given in seconds, and the sizes of hard disks, files, etc. are always given in bytes and not in kilobytes, kibibytes, blocks or any other confusion of units.

Please use these functions even if you don’t like the display so much. After all, this is then consistent for the user. And future versions of Checkmk may be able to change the display or even make it configurable for the user. Your check plug-in will then also benefit from this.

Following the detailed description of all of the display functions (render functions), you will find a summary of these in the form of a clear table.

Absolute time specifications (timestamps) are formatted with render.date() or render.datetime(). The specifications are always in seconds from January 1, 1970, 00:00:00 UTC — the so-called epoch time. This is also the format used by the Python function time.time(). The advantage of this representation is that it can be used to calculate very easily, for example, the duration of an operation if the start and end times are known. The formula is then simply duration = end - start. These calculations also work independently of the time zone, daylight saving time changes or leap years.

render.date() outputs only the date, render.datetime() adds the time. The output is done according to the current time zone for the Checkmk server that is running the check!

Examples:

Now please don’t be surprised that render.date(0) outputs 01:00 as the time instead of 00:00! This is because we are writing this manual in the time zone for Germany, which is one hour ahead of UTC standard time (at least during standard time, because, as you know, January 1 is not in (the European Summer) daylight saving time)

For timespan there is still the function render.timespan(). This produces a duration in seconds and outputs it in a human readable form. For larger time spans, seconds or minutes are omitted.

A frequency is effectively the reciprocal of time. The canonical unit is Hz, which is equivalent to once per second. A field of application is, for example, for the clock rate of a CPU:

Whenever memory, files, hard disks, file systems and the like are concerned, the canonical unit is the byte. Since computers usually organize such things in powers of two, e.g. in units of 512, 1024 or 65536 bytes, from the beginning it has been accepted that a kilobyte is not 1000 but 1024 bytes. This is in itself very practical, because in this way mostly round numbers came out. The legendary Commodore C64 had a 64 kilobyte memory and not 65,536.

Unfortunately, at some point hard disk manufacturers came up with the idea of specifying the sizes of their disks in 1000’s of units. Since the difference between 1000 and 1024 is 2.4% for each size, and these are multiplied, a 1 GB disk (1024 times 1024 * 1024) suddenly becomes 1.07 GB. That sells better.

This annoying confusion persists to this day and continues to cause errors. As a remedy, new prefixes based on the binary system were defined by the International Electrotechnical Commission. Accordingly, nowadays a kilobyte is officially 1000 bytes, and a kibibyte is 1024 bytes (2 to the power of 10). In addition one should say Mebibyte and Gibitbyte and Tebibyte (ever heard of these?). The abbreviations are (attention, here at once always i, instead of e!) KiB, MiB, GiB and TiB.

Checkmk adapts itself to this standard and helps you with multiple adapted render functions so that you can always produce correct outputs. So specifically for hard disks and file systems there is the render.disksize() function, which gives the output in powers of 1000.

For the sizes of files it is common to specify the exact size in bytes without rounding. This has the advantage that you can see very quickly if a file has changed even minimally or that two files are (probably) the same. The render.filesize() function is responsible for this:

If you want to output a size that is not a disk or a file size, just use the generic render.bytes(). This will give you the output in the classic 1024’s in the new official notation:

Networkers have their own terms and ways of expressing things. And as always, in each domain Checkmk tries hard to adopt the way of communicating that is customary there. That’s why there are three different rendering functions for data rates and speeds. All of these have in common that the rates are passed in bytes per second, even when the output is in bits!

render.nicspeed() represents the maximum speed of a network card or switch port. Since they are not measured values, there is no need to do any rounding. Although no port can send single bits, the specifications are in bits for historical reasons. Attention: you must however always pass bytes per second here as well!

Examples:

render.networkbandwidth() is for an actual measured transmission speed on the network. The input value is again bytes per second (or 'octets' as a networker would say):

Where the network is not involved and data rates are nevertheless output, bytes are again common. The most prominent examples are the IO rates of hard disks. For this there is the render function render.iobandwidth(), which in Checkmk works with powers of 1000:

The render.percent() function represents a percentage value — rounded to two decimal places. It is an exception to the other functions in that the actual natural value — that is, the ratio — is not passed here, but really the percentage. So if something is half full, for example, rather than 0.5. you must pass 50.

Because it can sometimes be interesting to know whether a value is almost zero or exactly zero, values that are greater than zero but less than 0.01 are marked by adding a "<" sign.

Here to recap is an overview of all of the render functions:

In one of our previous examples, we generated the state WARN if there were only 10 or fewer slots left. The number 10 was coded directly into the check function — hard-coded, as programmers would say. In Checkmk, however, users are more used to being able to configure such thresholds and parameters by rule. We will therefore take a look at how you can improve your check so that it can be configured via the setup interface.

To do this, we need to distinguish between two scenarios:

The rule sets supplied for the parameters of checks can be found in the lib/check_mk/gui/plugins/wato/check_parameters/ directory. Let’s take the file memory_simple.py as an example. This defines a rule set with the following section:

Here the important point for you is the keyword check_group_name, which is set here to 'memory_simple'. This establishes the connection to the check plug-in. You do this when registering the check with the keyword check_ruleset_name, for example:

The definition of default parameters via the keyword check_default_parameters is also mandatory. These parameters apply to your check if the user has not yet created a rule. If there are no mandatory parameters, you can simply take the empty dictionary {} as the value.

We will see below how the value configured by the user arrives at the check function.

If there is no suitable rule set (which is probably the usual situation), we will have to create a new one ourselves. To do this, we create a file in the local/share/check_mk/web/plugins/wato directory. The name of the file should be based on that of the check and, like all plug-in files, it must have the extension '.py'.

Let’s look at the structure of such a file step by step. First come some import commands. If you want the texts in your file to be translatable into other languages, import the _ (underscore) function. This is an identifier for all translatable texts. For example, you can code calling the function _("Threshold for warn") instead of "Threshold for warn". The translation system in Checkmk, which is based on gettext, finds such texts and includes them in the list of texts to be translated. If you only build the check for yourself, you can do without it and do not need the following import command:

Next we import the so-called ValueSpecs. A ValueSpec is a very practical and universal tool that uses Checkmk in many places. It is used to generate customised input masks, to display and validate the entered values and to convert them into Python data structures. In the following example, Dictionary, Integer and TextInput will be imported.

You will definitely need the Dictionary. Since version 2.0.0 of Checkmk it is mandatory that check parameters are Python dictionaries. Previously, it could also be a pair (a tuple of two numbers), e.g. Warn/Crit.

Integer is responsible for the input of a number without decimal places and TextInput for a Unicode text.

Next, symbols are imported that are needed for registration:

If your check does not have an item, import instead CheckParameterRulespecWithoutItem. About the RulespecGroup…​. we will explain more on this below.

Now come the actual definitions. First we define an input field with which the user can specify the item for the check. This is necessary for the rule condition and also for the manual creation of checks that are to function without discovery. We do this with TextInput. This is assigned a title via title, which is then usually displayed as a heading for the input field:

You can freely choose the name of the function that returns this ValueSpec as it is only required at the point further down. So that it does not become visible beyond the module boundary, it should begin with an underscore.

Next comes the ValueSpec for entering the actual check parameter. For this, as well, we create a function that generates it. The return Dictionary(…​) is mandatory. Within it, you create the list of sub-parameters for this check with elements=[…​]. In our example there is only one — the warning threshold for the free slots. This is required to be an integer, so we use integer here.

Last but not least, we now register a new rule set using the imported and self-defined items. For this purpose there is the rulespec_registry.register() function:

A few more notes on this:

In order for the rule to take effect, we must allow the check plug-in to accept check parameters and tell it which rule to use. To do this, the check_default_parameters entry must be present in the registration. In the simplest case, we pass an empty dictionary.

Secondly, we pass the check_ruleset_name to the registration function, i.e. the name we gave to the rule set above using check_group_name. This way Checkmk knows from which rule set the parameters are to be determined.

The whole thing will then look like this, for example:

Now Checkmk will try to pass parameters to the check function. For this to work, we need to extend the check function so that it expects the params argument as the second argument. This is inserted between item and section (If you build a check without an item, the item is of course omitted and params will be at the beginning):

It is highly recommended to now have the contents of the variable params printed out with a print command as a first test (or pprint if you want to have it a bit more convenient). Create different rules, and see which values arrive at params:

Very important: When everything is ready, be sure to remove the print commands again! These can otherwise disrupt the internal communication in Checkmk.

Now we adapt our check function so that the parameter passed can produce its desired effect. We get the value with the usually chosen key (here "warning_lower") from the parameters:

If a rule has been configured, we can now monitor the "free slots" in our example. However, if no rule has been defined, this check function will crash: Since the default parameters of the plug-in are not filled, in the absence of a rule the plug-in will generate a KeyError.

We can fix this problem by inserting a suitable parameter during registration:

You should always pass default values in this way (and not use the check plug-in to catch missing parameters), as these default parameters can also be displayed in the setup interface. For this purpose, there is e.g. the entry Show check parameters in the menu Display on the service configuration page for a host.

By the way, having a single value as a threshold is very uncommon in Checkmk. Since services can be in the states OK, WARN, CRIT, it is only logical to always define the parameters as tuples with two entries, i.e. as a pair of thresholds for WARN and CRIT. To do this, we adapt the rule set as follows:

Note that such a change of data type is an incompatible change: Existing rules can now no longer be loaded from the interface. And also the check function may run into problems if instead of an expected pair of numbers there is a single number in params. You can simply edit such rules. When you save them again, the new format will be used.

In Checkmk there are numerous ValueSpecs for all kinds of situations. Here are a few more useful ones:

In our example above, we have let the foobar plug-in generate the metric fooslots. Metrics are immediately visible in the Checkmk graphical interface without you having to do anything. A graph for each metric will be automatically generated in the service details.

However, there are a few limitations:

Before you do this, you should — as with the rule set for the parameters — first check whether Checkmk does not already come with a suitable metrics definition. The predefined metrics definitions can be found in the lib/check_mk/gui/plugins/metrics/ directory. For example, in the file cpu.py you will find a metric for the free space in a file system:

If this is suitable for your plug-in, you only need to use the name "util" in your call to the Metric() class. Everything else will then be automatically derived from it.

If there is no suitable metric, simply create one yourself. In our example we want to define our own metric for our fooslots. To do this, we create a file in local/share/check_mk/web/plugins/metrics:

Here are a few pointers:

This definition now ensures that the colour, title and unit of the metric are displayed according to our requirements.

If you want to combine several metrics in one graph — which is often very useful — you need, simply in the same file, a graph definition. This is done via the global dictionary graph_info.

For example, let’s assume our check has two metrics, fooslots and fooslots_free. The metric definitions would be, for example:

Now we add a graph that draws these two metrics as lines:

Notes on this:

If you would like to display a Perf-O-Meter in the service line in addition to our metric, you need another file, this time in the directory local/share/check_mk/web/plugins/perfometer.

Example:

Perf-O-Meters are a bit trickier than graphs because they have no legend. And that’s why it’s difficult with the range of values. Since the poor Perf-O-Meter cannot know which readings are even possible and the space is very limited, many built-in check plug-ins use a logarithmic representation. This is also the case in our example, half_value is the measured value that is displayed exactly in the middle of the Perf-O-Meter. With a value of 5, the bar would be half filled. And exponent describes the factor which is necessary so that another 10% of the range would be filled. So in this example, a reading of 10 would be displayed at 60% and one of 20 at 70%.

The advantage of this method is that when you have a list of services of the same type, you can quickly compare all Perf-O-Meters visually because they all have the same scale. And despite the very small representations, you can easily see the differences in both very small and large readings. The values are NOT to scale, however.

Alternatively, you can also use a linear Perf-O-Meter. This is always useful if there is a known maximum value. A typical situation would be measured values that represent percentages from 0 to 100. This would then look like this, for example:

There is another difference to the logarithmic representation — here segments is a list and allows multiple metrics to be displayed side by side.

As always, you can find examples in the many plug-ins supplied by Checkmk. These are also in the files in the directory lib/check_mk/gui/plugins/metrics.

The two functions saveint() and savefloat() have been dropped. As a reminder, saveint(x) returns 0 if x cannot be reasonably converted to a number, e.g. because it is an empty string or does not consist only of digits.

While there have been a few good use cases for this, it has been used incorrectly in the majority of cases, which in the past has resulted in many errors being obscured.

In a situation in which you want to get a 0 on an empty string — which is the most common 'good' use case of saveint(x) — you can simply code the following:

For savefloat() everything applies analogously.

If you look closely, you will see that these are nested lists. In the argument string_table you get a list which contains a list of words per line of the agent output. The lines are separated by sequences of spaces. Since our section contains only one word per line, the inner lists consist of only one entry each.

The following example makes the structure a little clearer:

The output will look like this:

For our example, we just need a simple list of device names, so we make our parse function unpack the single word from each line and package it into a nice new list:

The debug output then looks like this (please look carefully, there are now only a single pair of square brackets):

For the parse function to be complete, we now need to remove the debug message and — very importantly — deliver the new result with return:

Of course, from this point on, all of the relevant plug-ins must be able to work with the new data format.