Basic principles of monitoring with Checkmk

Everything in Checkmk revolves around hosts and services. A host can be many things, e.g.:

In monitoring a host always has one of the following states:

Alongside the state, a host has a number of other attributes that can be configured by the user, e.g.:

For the monitoring to be able to determine the UNREACH state, it must know which path it can use to contact each individual host. For this purpose, one or more so-called parent hosts can be specified for each host. For example, if a server A as seen from monitoring can only be reached via a router B, then B is a parent host of A. Only direct parents are configured in Checkmk. This then results in a tree-like structure with the Checkmk site in the middle (shown here as ):

Let’s assume that in the network topology example shown above, the hosts myhost and myhost4 are no longer reachable. The failure of myhost4 can be explained by the fact that myhost has failed. Therefore, myhost4 is classified as UNREACH in the monitoring. It is simply not possible to clearly determine why Checkmk can no longer reach myhost4, and the DOWN status would therefore be misleading in some circumstances. Instead, the UNREACH has the effect of suppressing notifications by default. This is after all the most important task of the parents concept, i.e. the avoidance of mass notifications in the event that an entire dependent network segment becomes unreachable for the monitoring due to an interruption at a single point.

The prevention of false alarms is also served by feature of the Checkmk Micro Core (CMC) used in the Checkmk Enterprise Editions. Here, the state change for a failed host is held back for a few moments and proceeds only when it is certain that the parent is still reachable. If, on the other hand, the parent is definitely DOWN, the host will switch to UNREACH — without a notification being triggered.

In some cases a host could have multiple parents, for example, when a router is running highly available in a cluster. It is sufficient for Checkmk to be able to uniquely determine the status of the host when one of these parents is reachable. Thus, when a host has multiple parents and at least one of these parents is UP, the host is considered reachable in the monitoring. In other words, in such a situation, the host will not automatically switch to the UNREACH state.

A host has a number of services. A service can be anything — please don’t confuse this with services in Windows. A service is any part or aspect of the host that can be OK, or not OK. Naturally the state can only be determined if the host is in UP state.

A service being monitored can have the following states:

When determining which condition is 'worse', Checkmk utilizes the following sequence:

OK → WARN → UNKNOWN → CRIT

A check ensures that a host or a service can be assigned a state. Which states these can be is described in the previous section. Services and checks are closely related. For this reason these terms are sometimes used interchangeably, perhaps even in this User Guide, although they are in fact different things.

In the Setup you can display which check plug-in is responsible for each service. Open the properties of a host with Setup > Hosts and then in the Hosts > Service Configuration menu the list of the services for this host. Then use Display > Show plugin names to display a new column which will show the check plug-in responsible for each service:

As you can see from the example of the df check plug-in, a check plug-in can be responsible for more than one service. By the way, the names of the check plug-ins listed in the column are also links that show a description of the check plug-in.

The connection and dependency of services and checks can also be seen in the monitoring. In the service list for a host in monitoring, you can see that in the action menu at the Reschedule entry, there is a yellow arrow for some services (), but a gray arrow for most of the others (). A service with the yellow arrow is based on an active check:

Such an active check is executed directly by Checkmk. Services with the gray arrow are based on passive checks whose data is fetched from another service, the Check_MK service. This is done for performance reasons and is a special feature of Checkmk.

Checkmk identifies every host that is not UP, and every service that is not OK as a problem. A problem can have two states: unhandled and handled. The procedure is that a new problem is first treated as unhandled. As soon as someone acknowledges (confirms) the problem it will be flagged as handled, and no surprise, unhandled problems are those which have not yet been attended to. The Overview in the sidebar therefore differentiates these two types of problems:

By the way: service problems from hosts that are currently not UP are not identified as problems.

Further details on acknowledgments can be found in its own article, Acknowledging problems.

When a host’s state changes, (e.g. from OK to CRIT), Checkmk registers a monitoring event. These events may or may not generate a notification. Checkmk is so designed that whenever a host or service has a problem, an email is sent to the object’s contacts (please note that the cmkadmin user, by default, is not a contact for any objects). These can be customized very flexibly however. Notifications also depend on a number of parameters. It is simplest when we look at cases for which notifications are not sent. Notifications are suppressed …​

If none of these prerequisites for suppressing notifications are satisfied, the monitoring core then creates a notification, which in a second step passes through a chain of rules. In these rules you can define further exclusion criteria, and decide whom should be notified and in what form (email, SMS, etc.)

All particulars concerning notifications can be found in its own article Notifications.

It sometimes happens that a service continuously and quickly changes its condition. In order to avoid continuous notifications, Checkmk switches such a service into the flapping state. This is illustrated with the icon. When a service enters a flapping state, a notification will be generated which informs the user of the situation, and silences further notifications. After a suitable time, if no further rapid changes are occurring, and a final (good or bad) status is evident, then the flapping status disappears and normal notifications resume.

If you perform maintenance work on a server, device or software, you will normally want to avoid potential problem notifications during this time. In addition, you will probably want to advise your colleagues that problems appearing in monitoring during this time may be temporarily ignored.

For this purpose you can enter a condition of scheduled downtimes on a host or service. This can can be done directly before starting the work, or in advance. Scheduled downtimes are illustrated by the icons:

While a host or service has a scheduled downtime:

Additionally, when you wish to later document statistics on the availability of hosts and services it is a good idea to include scheduled downtimes. These can be factored into later availability evaluations.

If you have been working with Checkmk for a while, it is possible that spider webs will be displayed in your host and service views. For services, for example, it looks like this:

These spider webs symbolize the stale state. Whenever there is a stale host or service, this will also be shown in the Overview snapin, which will be extended by the column Stale.

But what exactly does the stale state mean? In general, a host or service is marked as stale when Checkmk no longer receives up-to-date information about its status over a longer period of time:

You can adjust the time limit after which the hosts and services become stale. For this, read the section on check intervals.

You can restrict the time periods in which checks are executed. The rule sets Check period for hosts, Check period for active services and Check period for passive Checkmk services serve this purpose. Use these rules to select one of the available time periods as the check period.

Checks are executed at fixed intervals within state-based monitoring. Checkmk uses a default of one minute for service checks and 6 seconds for host checks with a Smart Ping.

These defaults can be overridden using the Normal check interval for service checks and Normal check interval for host checks rule sets:

If you now combine a check period with a check interval, you can ensure that an active check is executed precisely once a day at a very specific time. For example, if you set the check interval to 24 hours and the check period to 2:00 to 2:01 every day (i.e. only one minute per day), Checkmk will ensure that the check is actually moved to this short time window.

The state of the services will no longer be updated outside of this defined check period and the services will be marked as stale with the icon. With the global setting Staleness value to mark hosts / services stale you can define how much time should pass before a host/service goes to stale. This setting can be found under Setup > General > Global settings > User interface:

This factor represents n-times the check interval. So if your check interval is set to one minute (60 seconds), a service for which there are no new check results will go to stale after 1.5 times the time, i.e. after 90 seconds.

With the help of the check attempts option you can avoid notifications in the event of sporadic errors. This makes a check less sensitive, so to speak. You can use the rule sets Maximum number of check attempts for host and Maximum number of check attempts for service for this purpose.

If the check attempts are set to 3, for example, and the corresponding service becomes CRIT, then initially no notification is triggered. Only if the next two checks also produce a result that is not OK, will the count of the current attempts increase to 3 and the notification will be sent.

A service that is in this intermediate state — i.e. is not OK but has not yet reached the maximum number of check attempts — will have a soft state. Only a hard state will actually trigger a notification.