Special characteristics of the CMC

Host checks not only serve to trigger notifications in the case of a total host failure, but also to suppress notifications of service problems during the host’s down time. It is therefore important to quickly and accurately determine a host’s condition in the event of a service problem. Even if it has not been long since the last host check, this result can already be out of date. For this, it is sufficient when directly after a host failure, by chance the service check is executed first, rather than the host check. Even though it is down, the host would still be considered UP and the notification for the service would be sent — erroneously.

The CMC solves this problem very simply: if during a service problem the host is in an UP state, then the next host check will simply be waited-for. Due to the interval being very short at only six seconds, there is only a negligible delay to a notification — if the host is still UP and therefore the notification needs to be sent for the service. The CMC uses an additional trick — the icmpreceiver evaluates not only incoming ping responses, but also packets resulting from TCP connection establishments: SYN (synchronization) and RST (reset). Keep in mind, that the icmpreceiver will ignore each and every SNMP packet as SNMP does not communicate over the TCP protocol but over UDP.

As an example, let’s take the case of a check_http plug-in delivering a CRIT status, due to a queried webserver being unavailable. In this situation, following the start of the service check a TCP RST packet (connection refused) will be received from this server. The CMC therefore knows for certain that the host itself is UP and it can thus send the notification without delay.

The same principle is utilised when calculating network outages if parent hosts have been defined. Here as well notifications will at times be delayed briefly in order to wait for a verified status.

This procedure yields a number of advantages:

One disadvantage should not be hushed-up however: the Smart Ping host checks generate no performance data (packet runtimes). On hosts where these are required you can simply set up an active check via ping with the Check hosts with PING (ICMP Echo Request rule set.

In practice, not all hosts are checkable by ping. For these cases other methods in CMC can also be used for the host check, e.g., a TCP connection. Because these are generally exceptions they have no negative impact on the overall performance. The rule set here is Host Check Command.

There are firewalls that answer TCP connection packets to inaccessible hosts with a TCP RST packet. The trick is that the firewall is not permitted to register itself as the sender of this packet, rather the target host’s IP address must be specified. Smart Ping will view this packet as a sign of life and incorrectly assume that the target host is accessible.

In such a (rare) situation, via Global settings > Monitoring Core > Tuning of Smart PING you have the possibility of activating the Ignore TCP RST packets when determining host state option. Or, with check_icmp you can select a conventional ping as a host check for the affected hosts.

In order to avoid forking the core, during the program start the CMC creates a fixed number of very lean auxiliary processes whose task is to start the active check plug-ins: the check helpers. Not only do these fork much more quickly, but the forking also scales-up to cover all available cores because the core itself is no longer blocked. In this way, the execution of active checks (e.g. check_http) — whose runtimes are actually quite short — is greatly accelerated.

The CMC goes a significant step further however — because in a Checkmk environment active checks are rather an exception. Here Checkmk-based checks — in which only a single fork per host and interval is required — are primarily used.

To optimise the execution of these checks, the CMC maintains two other types of auxiliary processes: the Checkmk fetchers and the Checkmk checkers:

By default, 5 check helpers, 13 Checkmk fetchers and 4 Checkmk checkers are started. These values are set under Global settings > Monitoring Core and can be customized by you:

To find out if and how you need to change the default values, you have several options:

In the sidebar, the Core statistics snapin shows you the percentage utilisation averaged over the last 10-20 seconds:

For all auxiliary process types, there should always be enough processes to run the configured checks. If a pool is being 100% utilised, the checks will not be executed in time, the latency will grow and the states of the services will not be up-to-date.

The usage should not exceed 80% a few minutes after starting a site. For higher percentages, you should increase the number of processes. Since the necessary number of Checkmk fetchers grows with the number of monitored hosts and services, a correction is most likely here. However, be careful to create only as many auxiliary processes as are really needed, as each process occupies resources. In addition, all auxiliary processes are initialised in parallel when the CMC is started, which can lead to load peaks.

The Core statistics snapin shows you not only the usage, but also the latency. For these values, the simple rule applies: the lower, the better — and 0 seconds is therefore best.

Note: You can also display the values shown in the snapin for your site in the details of the OMD <site_name> performance service.

As an alternative to the Core statistics snapin, you can also have your configuration analyzed by Checkmk, with Setup > Maintenance > Analyze configuration. The advantage: Here you get an immediate evaluation from Checkmk how the state of the auxiliary processes is. Very handy is: If one of the auxiliary processes is not OK, you can open from the help text the corresponding Global settings option to change the value.