Special characteristics of the CMC

Host checks not only serve to trigger notifications in the case of a total host failure, but also to suppress notifications of service problems during the host’s down time. Service problems can arise and not be responsibility of the service itself, but rather of a failure condition of the host. It can happen that a host is actually DOWN even if its last known state in Checkmk is UP as per the last host check result. In such condition multiple service checks could returns problems that depend on the hosts DOWN state, resulting in sending service notifications — erroneously. It is therefore important to determine a host’s condition first, in the event of a service problem.

The CMC solves this problem very simply: if a service problem arises and the host is in an UP state, CMC will wait for the next host check. Due to the interval being very short at only (by default) 6 seconds, there is only a negligible delay to a notification — if the host is still UP and therefore the notification needs to be sent for the service.

As an example, let’s take the case of a check_http plug-in delivering a CRIT state, due to a queried web server being unavailable. In this situation, following the start of the service check a TCP RST packet (connection refused) will be received from this server by the icmpreceiver component. The CMC therefore knows for certain that the host itself is UP and it can thus send the notification without delay.

The same principle is utilized when calculating network outages if parent hosts have been defined. Here as well notifications will at times be delayed briefly in order to wait for a verified state.

This procedure yields a number of advantages:

One disadvantage should not be hushed-up however: the Smart Ping host checks generate no performance data (packet runtimes).

On hosts where these are required you can simply set up an active check via ping with the Check hosts with PING (ICMP Echo Request) rule set.

In practice, not all hosts are checkable by ping. For these cases other methods in CMC can also be used for the host check, e.g., a TCP connection. Because these are generally exceptions they have no negative impact on the overall performance. The rule set here is Host Check Command.

There are firewalls that answer TCP connection packets to inaccessible hosts with a TCP RST packet. The trick is that the firewall is not permitted to register itself as the sender of this packet, rather the target host’s IP address must be specified. Smart Ping will view this packet as a sign of life and incorrectly assume that the target host is accessible.

In such a (rare) situation, via Global settings > Monitoring Core > Tuning of Smart PING you have the possibility of activating the Ignore TCP RST packets when determining host state option. Or, with check_icmp you can select a conventional ping as a host check for the affected hosts.

In order to avoid forking the core, during the program start the CMC creates a fixed number of very lean auxiliary processes whose task is to start the active check plug-ins: the check helpers. Not only do these fork much more quickly, but the forking also scales-up to cover all available cores because the core itself is no longer blocked. In this way, the execution of active checks (e.g. check_http) — whose runtimes are actually quite short — is greatly accelerated.

The CMC goes a significant step further however — because in a Checkmk environment active checks are rather an exception. Here Checkmk-based checks — in which only a single fork per host and interval is required — are primarily used.

To optimize the execution of these checks, the CMC maintains two other types of auxiliary processes: the Checkmk fetchers and the Checkmk checkers.

By default, 5 check helpers, 13 Checkmk fetchers and 4 Checkmk checkers are started. These values are set under Global settings > Monitoring Core and can be customized by you:

To find out if and how you need to change the default values, you have several options:

In the sidebar, the Core statistics snap-in shows you the percentage utilization averaged over the last 10-20 seconds:

For all auxiliary process types, there should always be enough processes to run the configured checks. If a pool is being 100 % utilized, the checks will not be executed in time, the latency will grow and the states of the services will not be up to date.

The usage should not exceed 80 % a few minutes after starting a site. For higher percentages, you should increase the number of processes. Since the necessary number of Checkmk fetchers grows with the number of monitored hosts and services, a correction is most likely here. However, be careful to create only as many auxiliary processes as are really needed, as each process occupies resources. In addition, all auxiliary processes are initialized in parallel when the CMC is started, which can lead to load peaks.

The Core statistics snap-in shows you not only the usage, but also the latency. For these values, the simple rule applies: the lower, the better — and 0 seconds is therefore best.

As an alternative to the Core statistics snap-in, you can also have your configuration analyzed by Checkmk, with Setup > Maintenance > Analyze configuration. The advantage: Here you get an immediate evaluation from Checkmk how the state of the auxiliary processes is. Very handy is: If one of the auxiliary processes is not OK, you can open from the help text the corresponding Global settings option to change the value.