Tuning the monitoring core

Read the article on site services to find out which services you might be able to do without, and how to optimally configure the Apache site to suit the utilization behavior on your Checkmk server. If you have come across this article because of acute performance issues, reduce the number of Apache processes to the absolute minimum before applying the measures outlined here. You can then increase this back to a reasonable level later on—​once sufficient RAM and CPU resources have been freed up.

One area you should address before optimizing the monitoring core and core-related processes is the execution duration of agents. Between making contact with the agent and receiving the complete agent output, a process simply waits. In Checkmk Community, this is a single Python process that retrieves the agent output and then processes it. This usually requires 80 to 100 MB of resident memory. In the commercial editions, this is a so-called Fetcher, which does nothing other than pass on agent outputs. These, too, generally require 30 to 60 MB of resident memory.

To optimize performance, first determine the number of hosts monitored by regular agents, special agents and SNMP, and their average execution time. To help with this, take a look at the monitoring of 'typical' hosts—​the Check_MK service is of particular interest here. A top list is provided by Monitor > History > Service check durations; a complete list can be obtained via Livestatus.

The following table shows the waiting time for agent outputs and the duration of subsequent processing in a fictitious environment (using rather pessimistic assumptions regarding CPU time), totaled for a one-minute check interval. In practice, the assumed figures usually result in 3,000 to 5,000 services.

We will refer back to this table several times later in this article—​when calculating Checkers, Fetchers and concurrent Nagios checks.

As there are only 60 seconds in a minute, the table shows that at least 11 parallel processes are required which are simply waiting. Between 330 and 1100 MB of RAM are occupied during this time, but are not being put to good use. It is clear that simply optimizing the slowest hosts can have a positive impact on the performance of your Checkmk site.

We will not go into CPU time at this stage. However, as other processes and the overhead involved in creating processes have not yet been taken into account, a rough estimate suggests that two CPU cores will not be sufficient here. Operations should run satisfactorily on three cores, and with four cores there should still be scope for adding further hosts.

Possible measures for optimizing waiting times include:

Checkers are the auxiliary processes that process monitoring data before it is written to the monitoring core. Their task is, in practical terms, almost entirely CPU-intensive: The data to be processed is usually already in working memory, which is why no process waiting for I/O needs to be swapped out.

The usage of the checkers depends very heavily on the complexity of the check plug-ins being run and the volume of data being processed at any given time. It is therefore difficult to specify exact calculation methods. Ideally, you will have created a list of CPU times per Livestatus and totaled these. In this case, you simply need to divide the result by 60 (the regular check interval) and initially double it roughly.

If you have not carried out the calculations but would rather estimate and optimize: On current hardware, a checker can usually handle an average number of checks ranging from the four-digit to well into the five-digit range. A rough calculation of the number of checkers can therefore initially assume one checker per 5,000 services.

As checkers must store their configuration in working memory and the process size can therefore amount to several hundred megabytes, you should monitor the usage and promptly reduce the number of checkers back to a level where usage is around 50 %. Based on this usage, whilst monitoring latencies (particularly after a restart of the Checkmk site) and load peaks, you can further reduce the number of checkers (or include further services in the monitoring) until usage is around 80 %.

Fetchers are the auxiliary processes that retrieve agent output, launch special agents and query SNMP devices. Determining their number is somewhat more complex than that for checkers. It is not possible to make general statements about the number, as the requirements of different monitoring environments vary too greatly.

In the context of Agent execution table, we have already indicated that you can determine the number of fetchers required quite accurately based on the waiting time for agents. In addition to this rather theoretical approach, you can also identify the current usage and optimize the number of fetchers to suit the current usage situation.

In practice, you will want to combine both approaches: calculation and optimization.

For an initial Setup, you should aim for around 50 % fetcher usage. In the example shown above, with eleven fetchers required but only 13 available, you would be well over 80 % usage. It would therefore make sense to start by increasing the number of fetchers to around 20. The snap-in mentioned above Core statistics provides you with real-world feedback. If your calculations indicate only a very small number of required fetchers, do not go below three.

The following also applies to fetchers in the long term: Based on this usage, you can further reduce the number of fetchers (or include additional services in the monitoring) whilst observing latencies (particularly after a restart of the Checkmk site), behavior during network issues and load peaks. However, fetchers are more sensitive to external factors than checkers: If network latencies increase or packet loss occurs, the fetcher usage will rise significantly. You should therefore only aim for an upper level of 75 %–80 % if you have a very reliable network.

The auxiliary processes listed under Active check helpers are responsible for running active checks. The default setting for active checks in the CMC is a maximum of five running simultaneously. To help you estimate this, have another look at the list of results under Monitor > History > Service check durations. Here you will see that the inventory check (which runs once a day by default) can sometimes take up to a minute, a check_cert often completes in a few hundred milliseconds, and the values for check_httpv2--depending on the performance of the endpoint being checked—​vary considerably. Ultimately, you can perform a calculation as above--with five Active check helpers, 300 seconds are available per interval.

Alternatively, you can first take a look at the OMD mysite performance service, specifically the Helper usage metric and the Active check helper usage graph line there.

As with the fetchers, the following applies when calculating this figure: Do not go below three, and in larger environments, aim for an initial usage of around 50 %, which you can then gradually increase later.

A distinctive feature of the CMC is that changes to the configuration do not require the entire monitoring core to be restarted. The CMC can load a new configuration whilst the system is running and resume operation within fractions of a second. The settings for a full restart of the core are therefore more relevant during server updates and reboots.

Here, the default settings specify a 150-millisecond wait time between active checks and 1,200 milliseconds between retrieving agent output. This is far more generous than the default value the fetcher would otherwise allow. The reason for this is the expectation that other services will also be starting up whilst the server is restarting. For generously sized servers with many hosts under monitoring, this may mean that, although the load is low just a few minutes after the restart, Checkmk is still far from having updated all services.

Provided that the number of fetchers and checkers has already been optimized, you can significantly reduce these intervals in such situations.

Before you optimize forks, you should avoid them wherever possible. Bear in mind that calling the Check_MK service, which retrieves the agent’s output—​and then passes dozens of check results as passive checks directly to the monitoring core—​generates the same overhead in terms of process creation and result transfer as a simple active check that monitors exactly one network service. It therefore makes sense to reduce the number of active checks wherever possible. Possible examples include:

In the default configuration, running any number of parallel checks is possible. This setting works well under low load, from very small to very large systems. However, it is worth consistently adjusting this number to suit your own monitoring requirements, thereby achieving the best possible balance between memory usage and CPU utilization.

To calculate the value theoretically, divide the total execution time as shown in the table above by the check interval (default 60 seconds) and add a further 30 % to 50 % on top. Given the total time of 715 seconds calculated above, a value between 16 and 18 would therefore be a reasonable starting point from which you can continue to optimize.

If, after the core has started, as many services as possible are to be brought up to date as quickly as possible, a value must be found that ensures two things:

Given the 8:1 ratio mentioned above (715 seconds in total to 90 seconds of CPU time), a value of 32 for max_concurrent_checks would fulfill both conditions on a four-core system.

If you don’t want to do the initial calculations but are looking for a rule of thumb for the max_concurrent_checks value: Start—​based on your judgment—​with a value between two and four times the number of CPU cores, and fine-tune from there.

If, a good five minutes after restarting the monitoring core, you still have services that are Stale, you will need to increase max_concurrent_checks. Conversely, if no services are Stale after less than a minute, use htop to check whether spikes occur at the start of a check interval. When using htop, we recommend pressing the U (User) key to restrict the display to the site user and then pressing T (Tree) to switch to the tree view. If necessary, reduce the maximum number of concurrent checks further. However, even in small installations, do not go below two.

You must now also ensure that your Linux kernel permits the launch of all these processes.

For more than a quarter of a century, the Linux kernel has allowed memory overcommitment. This means that Linux allows processes to request a total amount of working memory greater than the physical memory available on a device. The default setting is to use a heuristic that attempts to predict future processes based on those started and terminated in the past. However, with Nagios, the ratio between working memory simply requested on a whim (“I might need this at some point…​”) and the memory actually used in resident memory is so high that the Linux kernel often refuses to create new processes unnecessarily.

You must therefore determine whether the virtual memory used by Nagios processes can be plausibly requested from the Linux kernel. If this is not the case, you must make changes to the memory overcommitment settings.

The settings shown here are particularly relevant if you have configured max_concurrent_host_checks to be unlimited or to a fairly high value.

By default, Nagios attempts to intelligently calculate the interval at which checks are initially started. However, this can lead to spikes at the start of the first check intervals, which persist for several minutes after the Checkmk site has started. In such cases, you can artificially deploy the initial checks over more than one check interval. Alternatively, if you wish to achieve full visibility in well under a minute, you can force an aggressive start behavior.

Assuming you identify 400 active checks (including Check_MK) for the 130 hosts assumed above, a value of 0.15 ensures that these fit exactly within the 60-second interval. For a smoother start and better distribution, you can also deploy the start over two or three check intervals.

The same applies to the variable host_inter_check_delay_method, which applies the same setting to host checks. The calculation here is simple: one check per host. With the figure of 130 hosts mentioned above, a reasonable value would be 0.4 seconds.

If you have set the number of max_concurrent_host_checks to a low value, you can use service_inter_check_delay_method=n to ensure that the initial scheduling takes place with maximum concurrency.

The actual Nagios core (scheduler, status calculation) runs as a single thread and therefore cannot benefit from multi-core processors. A fair amount happens in the main thread, and everything must take place sequentially. The following is relevant for Checkmk:

In Checkmk, the Python processes launched as active checks do the bulk of the work in retrieving and processing the agent data. These write the check results to the Nagios spool directory, from where they are read by the Reaper at one-second intervals.

We are aware of two potential bottlenecks:

The first issue mentioned can certainly occur on sites that use a high ratio of active checks to passive checks. The second is more likely to occur on larger-scale hardware with many cores and a very large number of passive checks.

Admins experienced in running Checkmk Community therefore often deploy the workload across multiple monitoring cores by setting up distributed monitoring on a single physical machine. A practical upper limit for such distributed monitoring is generally one Checkmk site per core.