Writing agent-based check plug-ins

Since Checkmk version 2.0.0, there has been a newly developed Check API for programming check plug-ins. We will show you how to use this Check API for plug-in programming.

You can access the Check API documentation at any time via the Checkmk user interface: Help > Developer resources > Check plugin API reference. In the new browser window, select BASE > Agent based API ("Check API") in the left-hand navigation bar:

Note for users of the Check API valid up to version 1.6.0: If you still have check plug-ins that were developed with the old API, you should migrate them to the new Check API soon. Although the old Check API will continue to be supported for a transitional period, this period will also come to an end. The advantages of the new Check API make up for the one-off migration effort, because it is more consistent, more logical, better documented and future-proof. In the blog post on the migration of check plug-ins we provide you with detailed information on the necessary steps for a migration.

If you are interested in programming check plug-ins, you will need the following:

There is always talk here of 'a check plug-in' and of writing it. But strictly speaking, you always need two plug-ins, the agent plug-in on the monitored host and the check plug-in on the Checkmk server. Both must be written and be compatible with each other — only then can each function smoothly in a monitoring operation.

The first step before writing any plug-in program is research! This means one needs to find out how to get the information you need for the monitoring.

For the example chosen, we use the fact that the Checkmk server is also the host. This means that initially it is sufficient to retrieve the status data via Livestatus, i.e. the data organized in tables that Checkmk holds about the monitored hosts and services in volatile memory.

Log in as an site user and query the information on the host groups with the following command:

The first line of the output contains the column names from the queried hostgroups table. The semicolon acts as a separator. The following lines then contain the contents of all columns, also separated by semicolons.

The output is already relatively confusing in this small example and contains information that is not relevant for our example. In general, you should leave the interpretation of the data to Checkmk. However, pre-filtering on the host can reduce the volume of data to be transferred if not all of it is actually required. So in this case, restrict the query to the relevant columns (Columns), to the names of the host groups (name) and the hosts in those groups (members):

The Livestatus interface expects to receive all commands and headers in their own separate line. The necessary line breaks are indicated by \n.

In this example, there are currently three host groups: two groups for the locations, and one for the check_mk group. This contains a host called localhost.

The check_mk host group is a special feature within the host groups. You have not created it yourself. And you cannot actively add a host to this group. So where does this host group come from? As by definition every host in Checkmk must belong to a group, Checkmk assigns every host that you do not specifically assign to a group to the 'special' check_mk group as the default.

As soon as you have assigned a host to one of your own host groups, it will be removed by Checkmk from the check_mk group. There is also no way to reassign a host to the check_mk host group.

Exactly these properties of the check_mk group are now used for our example: Since each host should be assigned to a location, the host group check_mk should be empty. If it is not empty, action is required, i.e. the hosts in it must be assigned to the host groups and thus to their appropriate locations.

Up until now, as a site user, you have used the lq command to display the information. This is helpful for getting an understanding of the data.

However, in order to retrieve this data from the Checkmk server, the new command must become part of the Checkmk agent on the monitored host. Theoretically, you could now directly edit the Checkmk agent in the /usr/bin/check_mk_agent file and include this part. This method would however have the disadvantage that your new command would disappear again when the agent software is updated because this file will be overwritten during the update.

It is therefore better to create an agent plug-in. All you need for this is an executable file that contains the command and which is located in the /usr/lib/check_mk_agent/plugins/ directory.

And one more thing is important: The data cannot be simply output. You will still need a section header. This is a specially formatted line that contains the name of the new agent plug-in. This section header allows Checkmk to later recognize where the data from the new agent plug-in begins and where the data from the preceding plug-in ends. It is easiest when the section header and check plug-in have the same name — even if this is not mandatory.

So first of all, you will need a meaningful name for your new check plug-in. This name may only contain lower case letters (only a-z, no umlauts, no accents), underscores and digits and must be unique. Avoid name conflicts with existing check plug-ins. If you are curious which names already exist, in a Checkmk site on the command line you can list these with cmk -L:

The output here only shows the first lines of the very long list. By using prefixes, the assignment of many check plug-ins can already be easily recognized here. The use of prefixes is therefore also recommended for your own check plug-ins. Incidentally, the second column shows how the respective check plug-in obtains its data.

A suitable name for the new check plug-in for our example is myhostgroups.

Now you have all of the information you need to create the script for the agent plug-in. Create a new file myhostgroups as the root user in the /usr/lib/check_mk_agent/plugins/ directory:

What does this mean in detail?

The first line contains the 'shebang' (this is an abbreviation for sharp and bang, the latter being an abbreviation for the exclamation mark), by which Linux recognizes that it should execute the script with the specified shell.

To keep the script adaptable, two variables are introduced next:

Use the echo command to output the section header. As the table columns are separated by a semicolon, use the addition sep(59) to specify that the semicolon is used as a separator for the data in the agent output. The 59 stands for the ASCII character code 59, the semicolon. Without this addition, the space character (ASCII character 32) would be used as a separator by default.

To be able to use the lq command, which is available to you as site user, in a script that is executed by the root user, prefix it with su.

Note: It is possible that accessing lq via su can cause problems. Alternatively, as a root user, you can also access Livestatus directly in the shell with printf or echo -e via a Unix socket. The article on Livestatus explains how to do this.

One more point is very important once you have created the file — make the file executable:

You can try out the agent plug-in directly by hand by entering the complete path as a command:

Host groups that do not contain any hosts are not listed here.

Testing and troubleshooting are the most important tasks when creating a functioning agent plug-in. It is best to proceed in three steps:

Testing the agent locally is very simple. As root, call the check_mk_agent command:

The new section must appear somewhere in the very long output. Agent plug-ins are output by the agent at the end of processing.

You can scroll through the output by appending less (press the space bar to scroll, / to search and q to exit):

Or you can search the output for the interesting lines. For example, grep with -A has an option to output a few more lines after each hit. This allows you to conveniently search and output the section:

The third and final test is then directly from the Checkmk site. Include the host in the monitoring (for example as localhost), log in as the site user and then retrieve the agent data with cmk -d:

This should produce the same output as the previous command.

If this works, your agent is ready. And what have you done for this? You have created a short script under the path /usr/lib/check_mk_agent/plugins/myhostgroups and made it executable.

Everything that follows now only takes place on the Checkmk server: There you write the check plug-in.

You will find a directory prepared for your own check plug-ins in the local hierarchy in the site directory. This is ~/local/lib/check_mk/base/plugins/agent_based/. Here in the path, base means the part of Checkmk that is responsible for the actual monitoring and notifications. The agent_based folder contains all of the plug-ins associated with the Checkmk agent (i.e. not notification plug-ins, for example). It is best to switch to this folder to work with:

The directory belongs to the site user and you can therefore edit it. You can edit your check plug-in with any text editor installed on the Linux system.

Thus, create the myhostgroups.py file for the check plug-in here. The convention is that the file name reflects the name of the agent section. It is mandatory that the file ends with .py, because from version 2.0.0 of Checkmk the check plug-ins are always real Python modules.

An executable basic framework (Download at GitHub), which you will expand step by step in the following, looks like this:

First you need to import the functions and classes required for the check plug-ins from Python modules. The simplest method for this is import *, but you should avoid this, as it obscures which namespaces have actually been made available.

For our example, only what is used in the rest of this article will be imported:

The parse function has the task of 'parsing' the 'raw' agent data, i.e. analyzing and splitting it up, and putting this data into a logically-structured form that is easy for all subsequent steps to process.

As shown in the section on testing the agent, the section supplied by the agent plug-in has the following structure:

Checkmk already splits the lines of the section supplied by the agent plug-in into a list of lines based on the separator in the section header (in the example ;), these lines in turn are lists of words. The following data structure is therefore available in Checkmk instead of the raw data from the agent plug-in:

In the inner list, the first element contains the name of the host group and the second the names of the hosts belonging to the group.

You can address all of this information, but only via its position in the data set. You would therefore always need to specify the number of square brackets and the 'sequence' number of the desired content(s) within each bracket. With larger volumes of data, this becomes increasingly complex and it becomes more and more difficult to maintain an overview.

At this point, the parse function offers clear advantages thanks to the structure it creates. It makes the code easier to read, the accesses are more performant and it is much easier to maintain an overview. It transforms the data structure supplied by Checkmk in such a way that you can address each of the individual values by name (or key) at will, and not be dependent on repeatedly searching through the field (array) to find what you are looking for:

The convention is that the parse function is named after the agent section and begins with parse_. It receives string_table as its only argument. Note that you are not free to choose the argument here — it really must be named like this.

With def you specify in Python that a function is to be defined below. parsed = {} creates the dictionary with the improved data structure. In our example, we will go through each line, element by element. The host group followed by the members of the host group is taken from each line and assembled into an entry for the dictionary.

The dictionary is then returned with return parsed.

Note: In the example shown above, you will find two commented-out lines. If you comment these later when testing the check plug-in, the data before and after executing the parse function will be displayed on the command line. This allows you to check whether the function really does what it is supposed to do.

In order for the whole thing to take effect, you must make the parse function, and the new agent section in general, known to Checkmk. To do this, call a registration function:

Here it is important that the name of the section exactly matches the section header in the agent output. From this moment on, every check plug-in that uses the myhostgroups section receives the return value of the parse function. As a rule, this will be the check plug-in of the same name. But other check plug-ins can also subscribe to this section, as we will show in the extension of the check plug-in.

By the way: If you want to know exactly how this works, you can take a look at the Check API documentation at this point. There you will find a detailed description of this registration function — and also of the functions and objects that will be used later in this article.

In order for Checkmk to know that there is a new check plug-in, this must be registered. It is done by calling the register.check_plugin function. You must always specify at least four things:

For the check plug-in it will look like this:

It is best not to try this out just yet, because you first need to write the functions discover_myhostgroups and check_myhostgroups, since these must appear in the source code before the above registration.

A special feature of Checkmk is the automatic discovery of services to be monitored. For this to work, each check plug-in must define a function that uses the agent output to recognize whether a service of this type or which services of this type should be created for the host in question.

The discovery function is always called when a service discovery is carried out for a host. It then decides whether or which services should be created. In the standard case, it receives exactly one argument with the name section. This contains the agent section data in a format prepared by the parse function.

Therefore, implement the following simple logic: If the myhostgroups agent section exists, then also create a suitable service. This will then automatically appear on all hosts on which the agent plug-in is deployed.

For check plug-ins that only create one service per host, no further information is required:

The discovery function must return an object of the service type for each service to be created using yield (not with return). In Python, yield has the same function as return — both return a value to the calling function. The decisive difference is that yield remembers how far the function has come in data processing. The next call continues after the last yield statement — and does not start at the beginning again. This means that not only the first hit is read out (as would be the case with return), but all hits in sequence (this advantage will become relevant later in our example with service discovery).

You can now move on to the actual check function, which uses the current agent output to decide which state the service should assume and can output further information.

The aim of the check function is to set up a check that can be used to verify whether a host group has been assigned for any host. To do this, it checks whether the check_mk host group contains hosts. If this is the case, the service should receive the status CRIT. If not, everything is OK and so is the state of the service.

Here is the implementation:

And now the explanation: The check_myhostgroups() function first fetches the value belonging to the check_mk key into the attr variable. Then the hosts variable is linked to the members value if it exists. If there are no members, hosts remains empty.

This is followed by an if query for the actual evaluation:

Once the check function has been created, the check plug-in is ready.

The check plug-in and as well as the agent plug-in have been made available on GitHub.

Testing and activation are carried out on the command line with the cmk command.

First try the service discovery with the option -I. By adding the v option (for verbose), detailed output is requested. The --detect-plugins restricts the command execution to this check plug-in and through localhost to this host:

As planned, the service discovery recognizes a new service in the myhostgroups check plug-in.

Now you can try out the check contained in the check plug-in:

By executing the check, the status of the service found previously will be determined.

If everything went as expected, you can activate the changes. If not, you will find helpful information in the chapter on troubleshooting.

Finally, activate the changes by restarting the monitoring core:

In the Checkmk monitoring, you will now find the new service Host group check_mk at the host localhost:

Congratulations on the successful creation of your first check plug-in!

The recently completed first check plug-in is now to be extended step by step. So far, the agent plug-in has only provided information on the names and members within the host groups. In order to be able to evaluate the status of the hosts and the services running on them, for example, more data is required.

In the example you have built a very simple check that creates a service on a host. However, it is also very common for there to be several services from a single check on a host.

The most common example of this is a service for a file system on a host. The check plug-in with the name df creates one service per file system on the host. In order to distinguish these services, the mount point of the file system (for example /var) or the drive letter (for example C:) is incorporated into the name of the service. This results in a service name such as Filesystem /var or Filesystem C:. The word /var or C: is referred to here as an item. We are therefore also talking about a check with items.

If you want to build a check with items, you must implement the following functions:

To test this in practice, you will create a separate service for each existing host group.

Since the myhostgroups check plug-in created in the previous chapter for checking the standard check_mk group should continue to function, this check plug-in remains as it is. For the extension, create a new check plug-in in the file myhostgroups.py — in the first step, in the same way as previously, by registering the plug-in.

Important: The new registration is made in addition to the existing one, the registration shown in the previous chapter remains unchanged. Here is just the new code:

So that the new check plug-in can be distinguished from the old one, it is given a unique name with myhostgroups_advanced. The sections parameter determines the sections of the agent output that the check plug-in subscribes to. Here, myhostgroups is used to specify that the new check plug-in uses the same data as the old one: the section of the agent plug-in prepared by the parse function. The service name now contains the placeholder %s. The name of the item is later inserted at this point by Checkmk. In the last two lines, the names for the new discovery function and the new check function are defined, both of which still need to be written.

First to the discovery function, which now has the task of determining the items to be monitored - this is also entered in addition to the existing one:

As with previously, the discovery function receives the section argument. The individual host groups are run through in a loop. All host groups are of interest here — with the exception of check_mk, as this special host group has already been taken care of by the existing myhostgroups check plug-in. Whenever an item is found, it is returned with yield, which creates an object of the type Service, that in turn receives the host group name as an item.

If the host is monitored later, the check function is called separately for each service — and therefore for each item. This brings you to the definition of the check function for the new myhostgroups_advanced check plug-in. The check function receives the item argument in addition to the section. The first line of the function then looks like this:

The algorithm for the check function is simple: If the host group exists, the service is set to OK and the number and names of the hosts in the group are listed. The complete function for this:

The check result is delivered by returning an object of the class Result via yield. This requires the parameters state and summary. Here, state defines the state of the service (in the example OK) and summary the text that is displayed in the Summary of the service. This is purely informative and is not evaluated further by Checkmk. You can find out more about this in next section.

So far, so good. But what happens if the item you are looking for is not found? This can happen if a service has already been created for a host group in the past, but this host group has now disappeared — either because the host group still exists in Checkmk but no longer contains a host, or because it has been deleted completely. In both cases, this host group will no longer be present in the agent output.

The good news: Checkmk takes care of this! If a searched for item is not found, Checkmk automatically generates the result UNKNOWN - Item not found in monitoring data for the service. This is intentional and a good thing. If a searched item is not found, you can simply run Python out of the function and let Checkmk do its work.

Checkmk only knows that the item that was there before is now gone. Checkmk does not know the reason for this — but you do. It is therefore appropriate to not keep such knowledge to yourself and to intercept this condition in the check function and to display a helpful message.

What has changed? The error condition is now handled first. Therefore, check in the if branch if the item really does not exist, set the status to CRIT and exit the function with return. In all other cases, return OK as before.

This means that you have adopted the disappearing host groups situation into the check function. Instead of UNKNOWN, the associated service will now be CRIT and contain information on the cause of the critical status.

This completes the new check plug-in as an extension of the old one. The extended agent plug-in and the extended file for the check plug-ins can again be found on GitHub. The latter contains the simple myhostgroups check plug-in from the previous chapter, the extended parse function and the components of the new myhostgroups_advanced check plug-in with its registration, discovery function and check function. Note that the functions must always be defined before registration so that there are no errors caused by undefined function names.

As the new myhostgroups_advanced check plug-in provides new services, you must perform a service discovery for this check plug-in and activate the changes in order to see these services in the monitoring:

Proceed as described in the simple check plug-in chapter.

In the monitoring of Checkmk, each service has a status — OK, WARN, etc.. — as well as a line of text. This text is located in the Summary column — as can be seen in the previous screenshot — and therefore has the task of providing a brief summary of the service’s status. The concept is that this text should not exceed a length of 60 characters. That ensures a concise table display without annoying line breaks.

There is also the Details field, in which all details on the status of the service are displayed, which also includes all of the summary information. Clicking on the service opens the service page, in which the two fields Summary and Details can be seen alongside many others.

When calling yield Result(...), you can determine which information is so important that it should be displayed in the summary, and for which it is sufficient for it to appear in the details.

In our example, you have always used a call of the following type:

This means that the text defined as summary always appears in the Summary — and also in the Details. You should therefore only use this for important information. If a host group contains many hosts, the summary list can become very long — longer than the recommended 60 characters. If a piece of information is of secondary importance, you can use details to specify that its text only appears in the details:

In the example above, the list of hosts is therefore only displayed in the Details. The Summary then only shows the number of hosts in the group:

In addition to summary and details, there is a third parameter. With notice you specify that a text for a service in the OK is only displayed in the details — but also in the summary for all other states. This makes it immediately clear from the summary why the service is not OK. The notice parameter is not particularly useful if texts are permanently bound to states, as in our example so far.

In summary, this means:

To prevent the number of services on a host from increasing excessively, several partial results are often combined in one service. For example, the Memory service under Linux not only checks RAM and swap usage, but also shared memory, page tables and all sorts of other things.

The Check API provides a very convenient interface for this. A check function can simply generate a result with yield as often as required. The overall status of the service is then based on the worst partial result in the order OK → WARN → UNKNOWN → CRIT.

In our example, use this option to define two additional results for each service of the host groups in addition to the existing result. These evaluate the percentage of hosts in the UP state and the services in the OK state. You use the additional columns of the host group table previously defined in the agent output and the parse function.

Now expand the check function in stages from top to bottom:

The if branch remains unchanged, i.e. the new partial results only apply to host groups that also exist. You then define five variables for the columns in the host group table contained in the section. On the one hand, this increases readability and, on the other hand, you can convert the strings read into numbers with int() for the four columns that are still to be used for the calculation.

The only existing result remains (almost) unchanged:

Only the access in the Python 'F-String' to the expression that returns the value is now easier than previously, since the attr is already in the variable definitions.

Now to the actual core of the extension, the definition of a result that implements the following statement: "The service of the host group is WARN when 90 % of the hosts are UP, and CRIT when 80 % of the hosts are UP." The convention here is that the check goes to WARN or CRIT as soon as the threshold is reached - and not only when it is exceeded. The Check API provides the auxiliary check_levels function for comparing a determined value with threshold values.

In the first line, the percentage is calculated from the total number and number of hosts in the UP state and stored in the hosts_up_perc variable. The single slash (/) executes a floating point division, which ensures that the result is a float value. This is useful because some of the functions used later on expect float as input.

In the second line, the result of the check_levels function is then returned as an object of the type result. This function is fed with the percentage just calculated as a value (hosts_up_perc), the two lower threshold values (levels_lower), a label that precedes the output (label) and finally with notice_only=True.

The last parameter makes use of the notice parameter already introduced in the previous section for the Result() object. With notice_only=True you specify that the text for the service is only displayed in the Summary if the status is not OK. However, partial results that lead to a WARN or CRIT will in any case always be visible in the summary — regardless of the value of notice_only.

Finally, you define the third result in the same way as the second, which evaluates the percentage of services in the OK state:

This completes the check function.

The service for a host group now evaluates three results and displays the worst status from these in the monitoring, as in the following example:

Not always, but often, checks deal with numbers — and these numbers are very often measured or calculated values. In our example, the number of hosts in the host group (num_hosts) and the number of hosts in the UP state (num_hosts_up) are the measured values. The percentage of hosts in the UP state (hosts_up_perc) is a value calculated from these values. If such a value can then be displayed over a time range, it is also referred to as a metric.

With its built-in graphing system, Checkmk has a component for storing, evaluating and displaying such values. This is completely independent of the calculation of the OK, WARN and CRIT states.

In this example, you will define the two calculated values, hosts_up_perc and services_ok_perc, as metrics. Metrics will be immediately visible in Checkmk’s graphical user interface without you having to do anything. A graph is automatically generated for each metric.

Metrics are determined by the check function and returned as an additional result. The easiest way is to add the metrics information to the check_levels() function in the call.

As a reminder, the line with the check_levels() function call from the previous section follows here:

The two new arguments for the metric are metric_name and boundaries:

To keep it nice and simple and meaningful, for the name of the metric use the name of the variable in which the percentage is stored as a value.

You can use boundaries to provide the graphing system with information about the range of possible values. This refers to the smallest and largest possible value. In the case of a percentage, the limits of 0.0 and 100.0 are not too difficult to determine. Both floating point numbers and integers (which are converted internally into floating point numbers) are permitted, but not strings. If only one limit of the value range is defined, simply enter None for the other, for example boundaries=(0.0, None).

With this extension, the check_levels function now also returns an object of the type Metric via yield in addition to the Result.

You can now define the metric services_ok_perc in the same way. The last lines of the check function will then look like this:

With the extended check function, both graphs are visible in the monitoring. In the service list, the icon now shows that there are graphs for the service. If you point to the icon with the mouse, the graphs will be displayed as a preview.

An overview of all graphs including their legends and more can be found in the service details.

But what do you do if the value for the desired metric has not been defined with the check_levels() function? You can, of course, define a metric independently of a function call. The Metric() object, which you can also create directly via its constructor, is used for this purpose. The alternative definition of a metric for the value hosts_up_perc looks like this:

The arguments in Metric() are very similar to those in the function call shown above: Mandatory are the first two arguments for the metric name and the value. In addition, there are two optional arguments: levels for the threshold values WARN and CRIT and boundaries for the value range.

Important: The specification of levels is only used here as information for displaying the graph. In the graph, the threshold values are usually drawn as yellow and red lines. The check_levels function with its specified threshold values is responsible for the actual checking.

Now use the option of defining not only the two calculated values, but all measured values as metrics using Metric() — in our example, the four measured values from the host group table. Limit yourself to the two obligatory specifications of metric name and value. The four new lines complete the extension of the check function for metrics:

This increases the number of graphs per service, but also gives you the option of combining multiple metrics in a single graph, for example. We show these and other options in the section Customizing the display of metrics below.

In the example file at GitHub you can find the entire check function again.

In the advanced check plug-in myhostgroups_advanced you have created the WARN state if only 90 % of the hosts are UP, and CRIT state if only 80 % of the hosts are UP. The numbers 90 and 80 are programmed directly into the check function or, as programmers would say, hard-coded. In Checkmk, however, users are used to being able to configure such threshold values and other check parameters via rules. For example, if a host group only has four members, then the two threshold values of 90 % and 80 % do not really fit well, since the percentage will drop to 75 % as soon as the first host fails and the status will go directly to CRIT — without an interim WARN level.

Therefore, the check plug-in should now be improved so that it can be configured via the Setup interface. To do this, you will need a rule set. If there is already a suitable rule set in Checkmk, you can use this. As the check function and rule set must match, this will not usually be the case. You can find more information on existing rule sets below.

Note: Be aware that the examples presented in this section are not covered by the Check API.

Before you create a new metric definition, you should first check whether Checkmk does not already have a suitable definition. The predefined metric definitions can be found in the ~/lib/check_mk/gui/plugins/metrics/ directory.

For CPU utilization, for example, you will find the metric with the name util in the cpu.py file:

If your check plug-in measures the CPU utilization as a percentage, you can use this metric definition without any problems. All you have to do is enter "util" as the name for the metric in your check function. Everything else will then be automatically derived from this.

If there is no suitable existing metric definition, simply create one yourself.

For our example, define your own metric for the number of services in the OK state. To do this, create a file in ~/local/share/check_mk/web/plugins/metrics:

Here is the explanation:

This definition in the metric file now ensures that the title, unit and color of the metric are displayed appropriately.

Analogous to the creation of a rule set file, the metrics file must first be read before the change becomes visible in the GUI. This is done by restarting the site’s Apache:

The metric graph will then look something like this in the Checkmk GUI:

If you want to combine several metrics into a single graph (which is often very useful), you will need a graph definition that you can add to the metrics file created in the previous section. This is done via the global dictionary graph_info.

For our example, the two metrics num_services_ok and num_services are to be displayed in the same graph. The metric definitions for this are as follows:

Now add a graph that shows these two metrics as lines:

The first entry under metrics determines which metric is used for the title of the graph. The result is the combined graph in the Checkmk GUI:

Would you like to display a Perf-O-Meter for a metric in the line in the service list? It could look like this, for example:

To create such a Perf-O-Meter, you need another file, this time in the ~/local/share/check_mk/web/plugins/perfometer directory:

Perf-O-Meters are somewhat trickier than graphs, as they have no legend. This makes it difficult to visualize the ranges of the displayed values. As the Perf-O-Meter cannot know which values are even possible and since the space is very limited, many built-in check plug-ins use a logarithmic representation. This is also the case in the example above:

The advantage with this method: If you equip a list of services of the same type with similar Perf-O-Meters, you can quickly compare the Perf-O-Meters with each other visually, as they all use the same scale. And despite the small display format, you can easily recognize the differences in both very small and very large values. The values in such a display are however not true to their actual scale.

The Perf-O-Meter above shows a metric, but not the one that is responsible for the status of the service. It is not the number of services in the OK state, but their percentage that determines the state of the host group service shown.

To display the metric of the percentage (services_ok_perc), you can use a linear Perf-O-Meter. This is always useful if there is a known maximum value. For example, it would look like this in the file:

And so in the GUI:

This is already an improvement. However, the state of the service is not only determined by the percentage of services in the OK state, but also by the percentage of hosts in the UP state. There are various ways to combine multiple metrics in a Perf-O-Meter. One method looks like this:

This ensures that the two metrics share the bar in the Perf-O-Meter and are displayed next to each other:

You can find many examples on this topic in the check plug-ins supplied by Checkmk — in the ~/lib/check_mk/gui/plugins/metrics/perfometers.py file.

Absolute time specifications (timestamps) are formatted with render.date() or render.datetime(). The information is always given as Unix time, i.e. in seconds from January 1, 1970, 00:00:00 UTC — the beginning of the Unix epoch. This is also the format used by the Python function time.time().

The advantage with this representation is that it is very easy to calculate, for example the calculation of a time range, when the start and end times are known. The formula is then simply duration = end - start. These calculations work regardless of the time zone, daylight saving time changes, or leap years.

render.date() only outputs the date, render.datetime() adds the time. The output is according to the current time zone in which the Checkmk server that is executing the check is located. Examples:

Do not be surprised that render.datetime(0) does not output 00:00 as the time, but 01:00. This is because we are writing this User Guide in the time zone for Germany — which is one hour ahead of standard UTC time (at least during standard time, because January 1st is not in daylight saving time).

For time ranges (or time spans) there is also the render.timespan() function. This is given a duration in seconds and outputs it in human-readable form. For larger time spans, seconds or minutes are omitted. If you have a time span in a TimeDelta object, use the total_seconds() function to read out the number of seconds as a floating point number.

A frequency is effectively the reciprocal of time. The canonical unit is Hz, which means the same as 1 / sec (the reciprocal of one second). An example of its use is the clock rate of a CPU:

Wherever working memory, files, hard disks, file systems and the like are concerned, the canonical unit is the byte. Since computers usually organize such things in powers of two, for example in units of 512, 1024 or 65,536 bytes, it became established right from the beginning, that a kilobyte is not 1000, and therefore a thousand times the unit, but 1024 (2 to the power of 10) bytes. This is admittedly illogical, but very practical because it usually results in round numbers. The legendary Commodore C64 had 64 kilobytes of memory and not 65.536.

Unfortunately, at some point hard disk manufacturers came up with the idea of specifying the sizes of their disks in units of 1000. Since the difference between 1000 and 1024 is 2.4 % for each size, and these are multiplied, a disk of size 1 GB (1024 times 1024 times 1024) suddenly becomes 1.07 GB. That sells better.

This annoying confusion still exists today and continues to produce errors. To alleviate this, the International Electrotechnical Commission (IEC) has defined new prefixes based on the binary system. Accordingly, today a kilobyte is officially 1000 bytes and a kibibyte is 1024 bytes (2 to the power of 10). In addition, one should say Mebibyte, Gibibyte and Tebibyte. The abbreviations are then KiB, MiB, GiB and TiB.

Checkmk conforms to this standard and helps you with a number of customized render functions to ensure that you always produce the correct output. For example, there is the render.disksize() function especially for hard disks and file systems, which produces its output in powers of 1000.

When it comes to the sizes of files, it is often customary to specify the exact size in bytes without rounding. This has the advantage that you can see very quickly if a file has changed even minimally or that two files are (probably) the same. The render.filesize() function is used for this:

If you want to output a value that is not the size of a hard disk or file, simply use the generic render.bytes(). With this you get the output in the 'classic' 1024 powers in the official notation:

Networkers have their own terms and ways of expressing things. And as always, Checkmk makes every effort to adopt the conventional way of communicating in each domain. That is why there are three different render functions for data rates and speeds. What these all have in common is that the rates are passed in bytes per second, even if the actual output is in bits!

render.nicspeed() represents the maximum speed of a network card or a switch port. As these are not measured values, there is no need for rounding. Although no port can send individual bits, the data is in bits for historical reasons.

Important: Nevertheless, you must also pass bytes per second here!

render.networkbandwidth() is intended for an actually measured transmission speed in the network. The input value is again bytes per second:

Where a network is not involved and data rates are still output, bytes are again common. The most prominent case is the IO rates of hard disks. The render.iobandwidth() function, which works with powers of 1000 in Checkmk, is used for this:

The render.percent() function represents a percentage — rounded to two decimal places. It is an exception to the other functions in that it does not pass the actual natural value — i.e. the ratio — but the real percentage. For example, if something is half full, you do not have to pass 0.5 but 50.

Because it can sometimes be interesting to know whether a value is almost zero or exactly zero, values are marked by appending a ‘<’ character, for values that are greater than zero but less than 0.01 percent.

In conclusion, here is an overview of all render functions:

If an exception occurs in monitoring or during service discovery in the Setup, the Summary contains references to the crash report that has just been created. It will look like this, for example:

Clicking on the icon displays a page with details in which you:

The traceback helps you as a developer to decide whether there is an error in the program (e.g. the call of a non-existent function) or agent data that could not be processed as expected. In the first case you will want to correct the error, in the second case it often makes sense to do nothing.

Submitting the report is of course only useful for check plug-ins that are officially part of Checkmk. If you make your own plug-ins available to third parties, you can ask your users to send you the data.

If you run your check plug-in using the command line, you will not receive any indication of the ID of any crash report generated. You will only see the summarized error message:

If you append the --debug option as an additional call parameter, you will receive the traceback from the Python interpreter:

If the error does not occur again the next time you call --debug, for example because a new agent output is available, you can also view the last crash reports in the file system:

There are two files in each of these folders:

In the examples shown above, we use the print() function to output the content of variables or the structure of objects for you as a developer. These functions for debug output must be removed from the finished check plug-in.

As an alternative to removal, you can also have your debug output displayed only when the check plug-in is called from the console in debug mode. To do this, import the debug object from the Checkmk toolbox and, if necessary, the pprint() formatting aid. You can now produce debug output depending on the value of the debug object:

Note that any remaining debug output should be used sparingly and limited to hints that will help subsequent users with debugging. Obvious and foreseeable user errors (for example, that the contents of the agent section indicate that the agent plug-in has been incorrectly configured) should be answered with the UNKNOWN state and include explanatory notes in the summary.

The question is how you should react if the output from the agent is not in the form you actually expect — whether from the Checkmk agent or received via SNMP. Suppose you always expect three words per line, what should you do if you only get two words?

Well, if this is a permitted and known behavior by the agent, then of course you need to intercept this and work with a case distinction. However, if this is not actually allowed, then it is best to act as if the line always consists of three words, for example with the following parse function:

If there is a line that does not consist of exactly three words, an exception is generated and you receive the very helpful crash report just mentioned.

If you access keys in a dictionary that are expected to be missing occasionally, it can of course make sense to react accordingly. This can be done by setting the service to CRIT or UNKNOWN and placing a note in the summary about the agent output that cannot be evaluated. In any case, it is better to use the get() function of the dictionary for this than to catch the KeyError exception. This is because get() returns an object of type None or an optional replacement to be passed as the second parameter if the key is not available:

What if the agent outputs correct data, but the item to be checked is missing? Like this, for example:

If the item you are looking for is not included, the loop is run through and Python simply drops out at the end of the function without returning a result via yield. And that is exactly the right approach! Because Checkmk recognizes that the item to be monitored is missing and generates the correct status and a suitable standard text with UNKNOWN.

If you want to simulate particular agent outputs, spool files are very helpful. You can use these to test borderline cases that are otherwise difficult to recreate. Or you can directly use the agent output that led to a crash report to test changes to a check plug-in.

First deactivate your regular agent plug-in, for example by revoking its execution authorization. Then create a file in the /var/lib/check_mk_agent/spool/ directory that contains the agent section (or expected agent sections) that your check plug-in expects, including the section header, and ends with Newline. The next time the agent is called, the content of the spool file will be transferred instead of the output from the agent plug-in.

With some check plug-ins that use items, it is quite possible that on larger servers several hundred services will be generated. If no separate parse function is used, this means that the entire list of hundreds of lines must be run through for each of the hundreds of items. The time required to search therefore increases by the square of the number of listed items, which means tens of thousands of comparisons for hundreds of services. If, on the other hand, the nested list is transferred to a dictionary, the time required to search for an element increases only linearly with the size of the dictionary.

In the Python wiki you will find a Overview of costs for searching in different data types, including an explanation and O notation. Using the parse function reduces the complexity of the search from O(n) to O(1).

As older versions of this article did not make use of the parse function, you should identify such check plug-ins and rewrite them to use a parse function.