Availability

Because Checkmk continuously monitors all hosts and services at regular intervals, it provides an exceptional basis for the later evaluation of their availability. And not only that — it can also be calculated for what percentage of a given time range an object was in a specified state or states, how often this state occurred, the duration of its longest stop, and much more.

Every calculation is based on a selection of objects and a specified time range in the past. Checkmk then reconstructs, within this time range, a chronological sequence of the states of all of the selected objects. Per state the times will be added together and displayed in a table. This table can, for example, show that a particular service experienced states of 99.95 % OK, and 0.005 % WARN over the defined time range.

When making such calculations, Checkmk also correctly takes account of such factors as scheduled downtimes, service downtimes, unmonitored time ranges and other special factors, enables the summary of states, and the ignoring of ‘brief interruptions’. Numerous customization options are also available. An availability of BI aggregations is also possible.

Through the inclusion of scheduled downtimes and similar special states, there is in theory a large number of possible combinations of states, for example: CRIT + In Planned Downtime + In Service Time + Flapping. Because most such combinations are not very useful, Checkmk reduces these to a small number and in doing so proceeds according to a principle of priorities. Since in the above example the service was in a scheduled downtime, the state in scheduled downtime simply applies, and the actual state is ignored. This reduces the list of possible states to the following:

This graphic shows the order in which states are prioritized. Later we will show how some states can be ignored or combined. Here are the states again, in detail:

Generating an availability analysis is very simple. First retrieve any view of hosts, services or BI aggregations. There you will find in the menu Services menu the Availability item, which takes you directly to the calculation of the availability of the selected objects. The data will be displayed as a table:

The table shows the same objects that were seen in the preceding view. Shown in each column is the proportion of the requested time range an object was in the state being queried. The value is given as a percentage, by default to two decimal places — which you can also easily customize.

The time range query can be customized with the Availability > Change display options > Time Range menu item. More on this later…​

You have the option of receiving the table as a PDF (commercial editions only). A download of the data in the CSV-Format is also possible (Export as CSV). It will look like this for the above example:

With the help of an automation user, which can authenticate itself via the URL, you can also retrieve data — e.g., with wget or curl) — and automatically process the data under script control.

The symbol can be found in every line of the table. This button takes you to a timeline display for the relevant object, in which it is precisely itemized which state change(s) occurred in the specified time range (abbreviated here):

Some useful tips:

As just mentioned, with the symbol the timelines offer the possibility of adding an annotation to a time period. A pre-filled form (‘Properties’) is provided into which comments can be entered:

With this you can define and extend the time range with different values as required. This is, for example, practical if you wish to annotate a larger time slice that has experienced repeated state changes. If you omit entering a service, the annotation will be created for the host. This will then be automatically related to all of the host’s services.

In every availability view all annotations applicable to the time range and the objects being displayed will be automatically visible.

But annotations have an additional function. Downtimes can be retrospectively entered — or conversely, removed. The availability analysis takes these corrections into account in exactly the same way as with ‘normal’ downtimes. There are at least two legitimate justifications for this:

To reclassify downtimes, simply select the Reclassify downtime of this period checkbox:

In the availability table, alongside the symbol for the timeline a further symbol can be found: . This takes you to a view of the monitoring history with a pre-filled filter for the relevant object and the time range for the query. Here you will not only see the event on which the availability analysis is based (the state changes), but also the associated notifications and similar events:

What is not visible here is the state of the object at the start of the query’s time period. The availability calculation looks even further back into the past in order to reliably determine the starting state reliably.

Many options influence the format of the displays, while others in turn influence the calculation methods. To begin with, let’s look at the displays:

The states described in the introduction can be customized and condensed in very many ways. In this way very different forms of evaluation can be flexibly generated. The are various options for this.

Sometimes monitoring often produce momentary problem messages, but under normal conditions the object is already OK by the time the next check runs (after one minute) — and no way has been found through adjusting thresholds or similar to get a neat grip on such cases. A common solution is to set the Maximum number of check attempts from 1 to 3 to allow more failures before a notification is triggered. Thus the concept of Soft states has been developed — meaning the WARN, CRIT or UNKNOWN states — as long as all of the permitted attempts have not been ‘used up’.

We are occasionally asked by users who use this feature why Checkmk’s Availability Module has no function for calculating using only Hard states. The reason for this: There is a better solution! One could use the hard states as the basis…​

The Short time Intervals option is much more flexible and at the same time very simple. Simply define a length of time which must be exceeded before the states will be evaluated.

Assume that the time value has been set to 2.5 minutes (150 seconds). If a service has been continuously OK, then is CRIT for 2 minutes, and then reverts to OK, the short CRIT-interval will simply be assessed as OK! The opposite situation incidentally also works! A short OK within a long WARN-phase will likewise be assessed as WARN.

Generally speaking, short intervals for which before and after the same state prevails will receive that same state. For a sequence of OK, then a 2 minute WARN, followed by CRIT, the WARN will persist — even if it was of a shorter duration than the defined length of time!

Bear in mind when defining the time, that in Checkmk the standard check interval is one minute. Thus every state has a duration of multiples of approximately one minute. Because the agent’s actual response times vary slightly, this can easily be 61 or 59 seconds. Therefore it is safer to not enter exact minutes for the value, rather to include a buffer — hence the example with 2.5 minutes.

An important function of the availability calculations in Checkmk is that they can be made dependent on time periods. With this times can be defined for every individual host or service. In these times the host/service will be expected to be available and the state then used for the calculations. Therefore every object has the Service period attribute. The procedure is as follows:

Here there are three simple possibilities. The default Base report only on service times hides times outside the defined service times. These hidden times then don’t count towards the 100 %. Only the time ranges within the service times will be actually considered. In the timeline display the remaining times will be ‘grayed-out’.

Base report only on non-service times performs the opposite, and and in effect calculates the inverse display: How good was the availability outside the service times?

The third option Include both service and non-service times deactivates the complete concept of service times and shows the calculations for all times from Monday 00:00 to Sunday 24:00.

By the way: If a host is not in the service time, for Checkmk it does not automatically mean that this also applies to the services on the host. Services always require their own rule in Service period for services.

When calculating availability, the complete history of the selected object must be reopened. How that works in detail can be learned further below. Especially in the Checkmk Raw Edition, the analysis can take some time, since its core has no cache for the required data and the text-based log data must be sequentially searched.

So that an excessively-complex query — that may possibly have been started unintentionally — does not tie up an Apache process, consume CPU and thus ‘hangs’, there are two options to limit the calculation’s duration. Both are activated by default:

The Query time limit limits the duration of the underlying query to the monitoring core to a specified time. This is predefined as thirty seconds. If this time is exceeded the analysis will be aborted and an error highlighted. If you are certain that the analysis can be allowed to run for longer, simply raise the timeout manually.

The Limit processed data option protects from calculations with many objects. Here a limit will be applied that functions analogous to that in the views. If the query to the monitoring core will produce more than 5000 time periods, the calculation will be aborted with a warning. The limitation will have been pre-processed in the core — where the data is gathered.

A powerful feature of Checkmk’s availability calculation is the facility to calculate the availability from BI aggregations. The big attraction here is that for this purpose Checkmk retroactively reconstructs the precise state of the respective aggregates at a particular point in time by using the protocols of the states of the individual hosts and services.

Why so much time and effort? Why not just query the BI aggregation with an active Check, and then show its availability? Well, the effort has quite a number of advantages for the user:

Retrieving the availability view is initially analogous to that for the hosts and services. Select a view with one or more BI aggregations, and select the BI Aggregations > Availability menu item. Here there is also a second method — every BI aggregation has a direct path to its availability using the symbol:

In itself the calculation is initially analogous to that for the services, however without the Host down and flapping columns, since these states do not exist for BI:

The big difference is in the time line view. The following example shows an aggregate in our demo server, which was CRIT for a very brief interval of one second (this would be a good example for the use of the Short time intervals option).

Do you want to know what the cause of the outage was? A simple click on the magic wand is enough. This enables a journey through time to the exact point of time when the outage occurred, and opens a display of the BI aggregation at that time — in the following image, already opened at the correct location:

For calculating the availability, Checkmk accesses the archived monitoring history logs, and to do so orients itself to the state changes. If, for example, at 5:14 p.m. on 10/20/2021 a service changes its state to CRIT, and then at at 5:24 p.m. changes back to OK, then you know that during this 10 minute time period the service was in a CRIT state.

These state changes are recorded in the Monitoring log, have the alert type HOST ALERT or SERVICE ALERT, and look like this for example:

There is always a current log file which includes entries for the most recent activity up to the present point in time, as well as a directory with an archive of the preceding periods. The location of these files varies, depending on the monitoring core in use:

The user interface does not access these files directly, rather it queries them using a Livestatus-query issued from the monitoring core. Among other factors, this is important since in a distributed monitoring the history files are not stored on the same system as the GUI.

The Livestatus query makes use of the statehist table. In contrast to the log table – which provides a ‘naked’ access to the history — here the statehist table is used because it has already performed the initial time-consuming calculation steps. Among other things it assumes the task of checking back in the history to determine the initial state, and the calculation of time periods with the same state, with their starts, durations and ends.

The condensing of the states procedure is performed in the user overview by the Availability Module, as described at the beginning of this article.