Basic principles of monitoring with Checkmk

Until now we have been concerned with the installation and implementation of Checkmk. Now it is time to begin explaining the basic concepts and definitions of monitoring (with Checkmk).

It is important to understand the basic differences between states and events - and namely for a very practical benefit. Most classic IT monitoring systems revolve around events. An event is something that occurs uniquely at a particular time. A good example would be error when accessing drive X. Typical sources of events are syslog messages, SNMP traps, the Windows Event Log, and log data entries. Events are quasi-spontaneous (self-generating, asynchronous) occurrences.

In contrast a state describes a sustained situation, e.g. drive X is online. In order to observe the state of something, the monitoring system must regularly poll it. As the example shows, in monitoring it is often possible to choose to work with events or with states.

Checkmk can accommodate both states and events, but, where the choice is available, always prioritize state-based monitoring. The reason for this lies in the numerous advantages of this method. Some of these are:

One can well say that Checkmk’s state-based monitoring is the norm. For the processing of states, the Checkmk Event Console is also available. This is specialized for the correlation and evaluation of large numbers of events and is seamlessly-integrated into the Checkmk platform.

Hosts and services can be grouped for an overview. In this way a host/service can be in more than one group. These groups are purely optional and not required for the configuration. Host groups can be useful when, alongside the folder structure in which the hosts are managed, an additional grouping is desired. If for example you have built a folder structure according to geographic standpoints, then it could be useful to have a Linux-Server host group e.g., that lists all Linux servers regardless of their geographic locations.

Contacts and contact groups offer the possibility of assigning persons to hosts and services. A contact correlates with a user name or web interface. The correlation with hosts and services does not occur directly however, rather via contact groups. Firstly, a contact (e.g. harri) is assigned to a contact group (e.g. linux-admins). Then hosts - or as required, individual services - can be assigned to the contact group. In this way users, and likewise hosts and services can be assigned to multiple contact groups.

These assignments are useful for a number of reasons:

By the way - the user cmkadmin, who is automatically defined by the creation of an instance, is always permitted to view all hosts and services even when they are not a contact. This is determined through their role as administrator.

Whereas the persons who are responsible or authorized for a particular host or service are defined through contacts and contact groups, their privileges are controlled via roles. Checkmk is supplied with three roles from which further roles can be later derived. Each role defines a series of rights which may be customized. The standard roles have the following meanings:

Time periods define regular, weekly-recurring time periods that are used in various positions in the monitoring’s configuration. A typical time period could be called work hours and could contain the time from 8:00 to 17:00 on all weekdays except Saturday and Sunday. The period 24X7 simply includes all times and is predefined. Time periods can also include exceptions for particular calendar days - e.g. Bavarian public holidays.

Some important situations which use time periods are:

The execution of checks occurs at fixed intervals in status-based monitoring. Checkmk uses one minute as its standard. Every check is therefore performed once per minute. This can be altered in the configuration:

Through defining a check period other than 24X7, the execution of active checks can be interrupted in specified time frames. The service’s status will no longer be updated, and will be flagged as stale, symbolized by .

In combination with a long check interval one can ensure that an active check is performed once per day at a specified time. If you set an interval of e.g. 24 hours and the check period at 02:00 - 02:01 on every day (only one minute per day), then Checkmk will ensure that the check really will be executed in this short time frame.

With the aid of max check attempts you can avoid notifications in the case of sporadic errors. In this way you are effectively making a check less sensitive. If the check attempts are set to e.g. 3, and the corresponding service becomes CRIT, then initially no notification will be generated. If the next two checks produce a result other than OK, the number of current attempts will increase to 3 and a notification will be sent.

A service that finds itself in this intermediate state - is thus not OK, but has not yet reached its maximum number of attempts - has a soft state.

If you look at the Checkmk interface you can see that for some services in the menu a yellow arrow () is shown, but a gray arrow () for most others. The services with the yellow arrow are active checks. These are executed by Checkmk directly. Services with a gray arrow are those for which the check results are determined by the active check Check_MK. These occur for performance reasons and illustrate a special feature of Checkmk:

In order that the target system (server, network device, etc.) is not newly-contacted for every single service, once per interval Checkmk collects all important data in one pass. From this data, in a single action it calculates new results for all passive checks. This conserves CPU resources on both systems and is an important factor that supports Checkmk’s high performance and scalability.

The following table provides a short overview of the most important status icons appearing beside hosts and services: