Distributed Monitoring

Livestatus is an interface integrated into the monitoring core which enables other external programs to query status data and execute commands. Livestatus can be made available over the network so that it can be accessed by a remote Checkmk-instance. Checkmk’s user interface uses livestatus to combine all tethered instances into a general overview. This then feels like a ‘large’ monitoring system.

The following diagram schematically shows the structure of a monitoring with livestatus distributed over three locations. The Checkmk-instance Central Site is found in the central processing site. From here central systems will be directly controlled. Additionally, there are the remote Site 1 and remote Site 2 instances which are located in other networks and controlled by their local systems:

What makes this method special is that the monitoring status of the remote instances is not sent continuously to the central site. The GUI always only retrieves data live from the remote instances when it is required by a user in the control centre. The data is then compiled into a centralised view. There is thus no central data holding, which means it offers huge advantages for scaling-up!

Here are some of the advantages of this method:

In a system distributed using Livestatus as described above, it is quite possible that the individual instances can be independently maintained by different teams, and the central site only has the task of providing a centralised dashboard.

In the case of multiple, or all instances needing to be administered by the same team, a central configuration is much easier to handle. Checkmk supports this and refers to such a configuration as a ‘distributed WATO’. With this all hosts and services, users and permissions, time periods, and notifications, etc., will be maintained centrally on the central using WATO, and then depending on their tasks, be automatically distributed to the remote instances.

Such a system not only has a common status overview but also a common configuration, and effectively ‘feels like a large system’.

No special requirements are placed on the central instance. This means that a long-established instance can be expanded into a distributed monitoring without requiring additional modifications.

The remote instances are then generated as new instances in the usual way with omd create. This will naturally take place on the (remote) server intended for the respective remote instance.

Special notes:

The most important step is now to enable live status via TCP on the network. Please note that live status is not per se a secure protocol and should only be used within a secure network (secured LAN, VPN, etc.). The enabling appears per omd config as an instance user on a stopped site:

Now select Distributed Monitoring:

Set LIVESTATUS_TCP to ‘on’ and enter an available port number for LIVESTATUS_TCP_PORT that is explicit on this server. The default is 6557:

After saving, start the instance as normal with omd start:

Retain the password for cmkadmin. Once the remote has been subordinated to the central instance, all users will likewise be replaced by those from the central instance.

The remote is now ready. Verify with netstat which should show that Port 6557 is open. The connection to this port is performed by an instance of the auxiliary daemon xinetd, which runs directly in the instance:

The configuration of the distributed monitoring takes place exclusively on the central instance. The required WATO-module is Distributed monitoring, and this serves to manage the connections to the individual instances. For this function the central instance itself counts as an instance and is already present in the list:

Using , now define the connection to the first remote instance:

In the Basic settings it is important to use the remote instance’s EXACT name – as defined with omd create – as the Site-ID. As always the alias can be defined as desired and also be later changed.

The Livestatus settings determine how the central queries the status of the remote instances via live status. The example in the screenshot shows a connection with the Connect via TCP method. This is the optimal for stable connections with short latency periods (such as, eg. in a LAN). We will discuss the optimal settings for WAN connections later.

The URL prefix is required for integrating other applications (e.g. PNP4Nagios). We will come to this subject separately later. Enter the HTTP-URL to the remote’s web interface here (only the part preceeding the check_mk/ component). If you basically access Checkmk per HTTPS, then substitute the http here with https. Further information can be found in the online help or the corresponding article regarding HTTPS together with Checkmk.

The use of Distributed WATO is, as we discussed in the introduction, optional. Activate this if you wish to configure the remote with and from the central instance. In such a case select the exact settings as shown in the image above.

A correct setting for the Multisite-URL of the remote site is very important. The URL must always end with /check_mk/. A connection with HTTPS is recommended, provided that the remote instance’s Apache supports HTTPS. This must be installed manually on the remote at the Linux level. For the Checkmk Appliance, HTTPS can be set up using the web-based configuration interface. If you utilise a self-signed certificate, you will require the Ignore SSL certificate errors check box.

Once the mask has been saved a second instance will appear in the overview:

The (so far) empty remote instance’s monitoring status is now correctly integrated. A Login to the remote’s WATO is still required for the distributed WATO. To this end, via HTTP the central instance exchanges a randomly-generated password with the remote instance, through which all future communication will take place. The cmkadmin access on the remote instance will subsequently no longer be used.

To login use the access data cmkadmin and the according password of the remote instance:

A successful login will be so acknowledged:

Should an error occur with the login, this could be due to a number of reasons – for example:

Points 1. and 2. can be easily tested by manually calling the remote’s URL in your browser.

When everything has been successful run Activate Changes. This will, as always, bring you to an overview of the not yet activated changes. Simultaneously it will also show the states of the livestatus connections, likewise the WATO-synchronisation states of the individual instances:

The Version column shows the Livestatus-version of the respective site. When using the CMC as the Checkmk’s core (Enterprise Editions), the core’s version number (shown in the ‘Core’ column) is identical to that of the livestatus. If you are using Nagios as the core ( Checkmk Raw Edition), the Nagios version number will be seen here.

The following symbols show WATO’s replication status:

In the Status column the state of the livestatus connection for the respective instance can be seen. This is shown purely for information since the configuration is not transmitted via Livestatus, but rather over HTTP. The following values are possible:

Clicking on the button will now synchronise all instances and activate the changes. This is performed in parallel, so that the overall time equates to the time required by the slowest instance. Included in the time is the creation of a configuration snapshot for the respective instance, the transmission over HTTP, the unpacking of the snapshot on the remote instance, and the activation of the changes.

Important: Do not leave the page before the synchronisation has been completed on all instances – leaving the page will interrupt the synchronisation.

Once your distributed environment has been installed you can begin to use it. You actually only need to tell each host by which instance it should be monitored. The central instance is specified by default.

The required attribute for this is ‘Monitored on site’. You can set this individually for each host. This can naturally also be performed at the folder level:

Adding hosts functions as usual – apart from the fact that the surveillance as well as the service discovery will be run from the respective remote instance, there are no special considerations.

When migrating hosts from one instance to another there are a couple of points to be aware of. Neither current nor historic status data from the host will be carried over. Only the host’s configuration is retained in the WATO. In effect it is as if the host has been removed from one instance and freshly-installed on the other instance:

If the continuity of the history is important to you, when implementing the monitoring you should carefully plan which host is to be monitored, and from where.

To achieve the encryption Checkmk uses the stunnel program along with its own certificate and its own Certificate Authority (CA) to sign the certificate. These will be individually generated automatically with a new instance and they are therefore not predefined static CAs or certificates. That is a very important safety factor to prevent fake certificates from being used by attackers, because any attackers could then gain access to a publicly-available CA.

The generated certificates also have the following properties:

The fact that the standard certificate is valid for so long very effectively prevents you from getting connection problems that you cannot classify. At the same time it is of course possible that once a certificate has been compromised it is accordingly long open to abuse. So if you fear that an attacker will gain access to the CA or to the instance certificate signed with it, always replace both certificates (CA and instance)!

In larger environments you might in any case want to use your own certificates. To replace the supplied ones, simply substitute the instance certificate with your own, and make sure that the CA which has signed the new certificate is also trusted.

For compatibility reasons the LIVESTATUS_TCP_TLS option will not be automatically activated after an update from an older version to 1.6.0, since in the new version it is only possible to use the connection with encryption. After the update, to make use of the new feature in your monitoring instances, stop the instance and activate the option mentioned:

Since the certificates were generated automatically during the update, the instance then immediately uses the new encryption feature. So that you can still access the instance from the central instance, in the second step activate the Encryption option in the Instance Connection Properties under WATO > Distributed Monitoring:

The last step is as described above – again here you first have to mark the CA of the remote instance as trusted.

All accesses of a monitored host are consistently carried out from the instance to which the host is assigned. This applies not only to the actual monitoring, but also to the service discovery, the Diagnostics page, the Notifications, Alert handlers and everything else. This point is very important as it is not assumed that the central instance actually has access to this host.

Some of the standard views are grouped according the instance from which the host will be monitored – this applies for, e.g. All hosts:

The instance will likewise be shown in the host’s or service’s details:

This information is generally available for use in a column when creating your own views. There is also a filter with which a view of hosts on a specific site can be filtered:

There is a Site status snap-in element for the side bar which can be added using . This displays the status of the individual instances, and it also provides the option of clicking on the status to temporarily hide or show individual sites. These will be flagged with the status. With this you can also disable a instance that is generating timeouts, thus avoiding superfluous timeouts:

This is not the same as disabling the livestatus connection using the connection configuration in WATO. Here the ‘disabling’ only affects the currently logged-in user and has a purely visual function. Clicking on an instance’s name will display a view of all of its hosts.

In a distributed monitoring the central instance control element has a different appearance. Each instance has its own global switch:

If you monitor with Checkmk HA-Cluster, the cluster’s individual nodes must be assigned to the same instance as the cluster itself. This is because determining the clustered services’ status accesses cache files generated through monitoring the node. This data is located locally on the respective instance.

Some check plug-ins use ‘Piggyback’ data, for example, for allocating monitoring data retrieved from an ESX-host to the individual virtual machines. For the same reason as with cluster monitoring, in distributed monitoring the ‘piggy’ (carrying) host as well as its dependent hosts must be monitored from the same instance. In the case of ESX this means that the virtual machines must be assigned to the same site in Checkmk as the ESX-System from which the monitoring data is collected. This can mean that it is better to poll the ESX-host system directly rather than to poll a global vCenter. Details for this can be found in the documentation on ESX-monitoring.

The Checkmk Hardware/Software inventory also functions in distributed environments. In doing so the inventory data from the var/check_mk/inventory directory must be regularly transmitted from the remotes to the central instance. For performance reasons the user interface always accesses this directory locally.

In the Checkmk Enterprise Editions the synchronisation is carried out automatically on all sites that are connected using the Livestatus proxy.

If you run inventories using the Checkmk Raw Edition in distributed systems, the directory must be regularly mirrored to the central instance with your own tools (e.g., with rsync).

Even when all instances are being centrally monitored, a login on an individual instance’s interface is quite possible and often also appropriate. For this reason WATO ensures that a user’s password is always the same for all sites.

A password change made by the administrator will take effect automatically as soon as it is shared to all instances with Activate Changes.

A change made by a user themselves using the sidebar in their personal settings works somewhat differently. This cannot execute an Activate changes since the user of course has no general authority for this function. In such a case WATO will automatically share the changed password across all instances – directly after it has been saved in fact.

As we all know, networks are never 100% available. If an instance is unreachable at the time of a password change, it will not receive the new password. Until the administrator successfully runs an Activate changes, or respectively, the next successful password change, this instance will retain the old password for the user. A status symbol will inform the user of the status of the password sharing to the individual instances.

It is self-evident that the host tags used in the remote must also be known to the central instance in order that they can be carried over. Check these before the migration and add any missing tags to the central instance manually. Here it is essential that the Tag-IDs match – the tag’s title is irrelevant.

Next, move the hosts and rules into the central WATO on the central instance. This only works for hosts and rules in sub-directories (i.e., not in the ‘Main directory’ ). Hosts in the main directory should first simply be moved into a remote instance’s sub-directory using WATO.

The actual migration can then be achieved quite simply by copying the appropriate directories. Each host directory in WATO corresponds to a directory within etc/check_mk/conf.d/wato/. These can be copied using a tool of your choice (e.g. scp) from the tethered site to the same location in the central instance. If a directory with the same name already exists there, simply rename it. Please note that Linux users and groups are also used by the central instance.

Following the copying the hosts should appear in the central’s setup – as well as the rules you have created in these folders. The folders’ characteristics will also be included with the copying. These can be found in the folder in the hidden .wato file.

So that the attributes of the central instance’s parent folder’s functions are correctly inherited, as a final step following the migration the parent folders’ characteristics must be opened and saved once – the host’s attributes will thereby be freshly defined.

The first method is very simple to implement. For the relevant sites, configure the URL-prefix in the connection’s attributes, and set it to the URL used for accessing this instance – though without the /check_mk/:

Checkmk will embed the graphs in the GUI so that the browser can retrieve the graphs’ PNG-images, or respectively, the website’s Iframes from PNP4Nagios over this URL. Specify the URL thus as it functions with the application’s browser. An access to the remote from the central instance is not necessary.

The URL method as just described is quick and easy to set up, but it has a few small disadvantages:

The best solution to this problem is to retrieve the PNP4Nagios-data from the central instance, rather than from the user’s browser itself. To this end, create a proxy rule on the central’s Apache-server. This will route PNP4Nagios queries per HTTP or HTTPS to the correct remote server. Important: this must be done on the operating system’s Apache, not that running on the instance. For this reason a root-permission is required.

The prerequisite for this setup is that all Checkmk instance-IDs in your network are explicit, since Apache must use the remote-ID to decide which server it should forward to.

Assuming the following example:

In the connection settings, now simply set /remote1/ as the URL-prefix:

With this, queries to PNP4Nagios initially go to the central instance on the /remote1 URL. Should the remote1 instance coincidentally be running on the same server as the central instance, you will now be finished and no proxy rule will be required, since the data can be delivered directly.

In the general case that the remote instance runs on another host, you will require the root-permission and must create a configurations file for the system-wide Apache server. The path for this file will depend on your Linux distribution:

The file consists of five lines for each tethered remote instance. In the following example, substitute the instance name (here remote1) and the instance’s URL (here 10.1.1.133/remote1/). Please note that for Apache it is relevant whether a URL ends with a (/) ‘slash’ or not:

This rule tells Apache that all URLs beginning with /remote1 are to be retrieved via reverse-proxy from the URL 10.1.1.133/remote1.

Important: don’t forget to activate the configuration. For SLES, Debian and Ubuntu, perform this with:

RedHat and CentOS require:

If everything has been done correctly, PNP4Nagios must now be able to access the graphs.

Installing the livestatus proxy is very simple. It is activated by default in the CEE – which can be seen when starting a site:

Select the setting ‘Use Livestatus Proxy-Daemon’ for the connection to the remote instances instead of ‘Connect via TCP’:

The details for host and port are as always. No changes must be made on the remote instance. In Number of channels to keep open enter the number of parallel TCP-connections the proxy should establish and sustain to the target site.

The TCP-connections pool is shared by all GUI enquiries. The number of connections limits the maximum number of queries that can be processed concurrently. This indirectly limits the number of users. In situations in which all channels are reserved this will not immediately lead to an error. The GUI waits a given time for a free channel. Most queries actually require only a few milliseconds.

If the GUI must wait longer than Timeout waiting for a free channel for a channel, it will be interrupted with an error and the user will receive an error message. In such a case the the number of connections should be increased. Be aware however that on the remote instance sufficient parallel incoming connections must be allowed – this is set to 20 by default. This setting can be found in the global options under Monitoring core > Maximum concurrent Livestatus connections.

The Regular heartbeat provides a constantly active monitoring of the connections directly at the protocol level. In the process the proxy regularly sends a simple Livestatus query which must be answered by the remote instance within the predetermined time (default: 2 seconds). With this method a situation where the target server and the TCP-port are actually reachable, but the monitoring core no longer responds, will also be detected.

If a response fails to appear, all connections will be declared ‘dead’, and following a ‘cooldown’ time (default: 4 seconds) will be newly established. All this takes place proactively – i.e. without a user needing to open a GUI-window. In this way outages can be quickly detected, and via a recovery the connections can be immediately reestablished and in the best case be available before a user even notices the outage.

The Caching ensures that static queries need only be responded-to once by the remote instance, and from that point of time can be responded to directly and locally, without delay. An example of this is the list of monitored hosts required by Quicksearch.

The Livestatus proxy has its own log file which can be found under var/log/liveproxyd.log. On a correctly-configured remote instance with five channels (standard) it will look something like this:

The Livestatus proxy regularly records its state in the var/log/liveproxyd.state file:

And when an instance is currently stopped the state will look like this:

Here the state is ‘starting’. The proxy is thus attempting to establish connections. There no channels yet. During this state queries to the site will be answered with an error.

First, on the remote instance, create a copy of its host’s and service’s configurations in Nagios-configuration format. Also redirect the output from livedump -TC to a file:

The start of the file will look something like this:

Transmit the file to the central instance, (e.g. with scp) and save them there in the ~/etc/nagios/conf.d/ directory – here Nagios expects to find the configuration data for hosts and services. Select a file name that ends with .cfg, for example ~/etc/nagios/conf.d/config-remote1.cfg. If an SSH-access from remote to central instance is possible it can be done, for example, as below:

Now log in to the central instance and activate the changes:

Now all of the remote’s hosts and services should appear in the central instance – initially with the PEND state, which they will retain for the time being:

Note:

Once the hosts are visible in the central instance, we will need to setup a (regular) transmission of the remotes' monitoring status. Again create a file with livedump, but this time without secondary options:

This file contains the states of all hosts and services in a format which Nagios can read directly from check results. The start of this file looks something like this:

Copy this file to the central instance into the ~/tmp/nagios/checkresults directory. Important: This file’s name must begin with c and be seven characters long. With scp it will look something like this:

Finally, create an empty file on the central instance with the same name and the .ok extension. With this Nagios will know that the status file has been transferred completely and can now be read in:

The status of the remotes’ hosts/services will now be immediately updated on the central instance:

The transmission of the status must from now on be made regularly. Livedump unfortunately doesn’t support this task and you will need to script it yourself. The livedump-ssh-recv script can be found in ~/share/check_mk/doc/treasures/livedump, which you can employ in order to receive Livedump updates (including those from the configuration) on the central instance per SSH. Details about this can be found in the script itself.

The configuration and staus dump can also be restricted by using Livestatus filters. For example, you could limit the hosts to the members of the mygroup hostgroup:

Further information on Livedump – in particular how to transfer the data via encrypted email – can be found in the README file in the ~/share/doc/check_mk/treasures/livedump directory.

The Checkmk Enterprise Editions provide a built-in mechanism for centralised notifications which can be individually activated for each remote instance. Such remotes then route all notifications to the central instance for further processing. The centralised notification is thereby independent of whether the distributed monitoring has been set up in the standard way, or with CMCDump, or by using a blend of these procedures. Technically speaking, the central notification server does not even need to be the ‘central’. This task can be taken on by any Checkmk instance.

If a remote instance has been set to ‘forwarding’, all notifications wiil be forwarded directly to the central instance as they would be from the core – effectively in a raw format. Once there the notification rules will be evaluated which actually decide who should be notified and how. The required notification scripts will be invoked on the central instance.

The first step for implementing centralised notification is to activate the notification spooler (mknotifyd) on all participating instances. This is an auxiliary process that is required on the central as well as on the remote instances. In newer Checkmk-versions the notification spooler is automatically aktivated. Please verify this with omd config and activate it if needed. This point can be found under Distributed Monitoring > MKNOTIFYD.

An omd status must show the mknotifyd process:

Only when the notification spooler is active will the point Notifications > Notification spooling be found under the global settings in WATO.

The remote and (notification-)central notification spoolers communicate with each other via TCP. Notifications are sent from remote to central instance. The central acknowledges to the remotes that the notifications have been received, which prevents notifications being lost even if the TCP connection is broken.

There are two alternatives for the construction of a TCP-connection:

Consequently there is nothing standing in the way of forwarding notifications if for network reasons establishing connections is only possible in a specific direction. The TCP-connections are supervised by the spooler with a heartbeat signal and are immediately reestablished as needed – not only in the event of a notification.

Since remote and central instance require different global settings you must make site specific settings for all remotes. Configuring the central instance is performed using the normal global settings. This is due to Checkmk currently not supporting any specific settings for the local instance (= central instance). Please note – these settings will be automatically inherited by all remotes for which no specific settings have been defined.

Let us look first at an example where the central instance establishes the TCP-connections to the remote instances.

Step 1: On the remote instance, edit the instance specific global setting Notifications > Notification Spooler Configuration and activate Accept incoming TCP connections. TCP-Port 6555 will be recommended for incoming connections. If there are no objections, adopt these settings.

Step 2: Now, likewise, in the Notification Spooling submenu only on the remote instance, select the option Forward to remote site by notification spooler.

Step 3: Now, on the central instance – i.e. in the normal global settings – configure the connection to the remote (and then to additional remotes as needed):

Step 4: Set the global setting Notification Spooling to Asynchronous local delivery by notification spooler, so that the central instance’s communications will also be processed over the same central spooler.

Step 5: Activate the changes.

If the TCP-connection should be established from the remote outwards, the procedure is identical, differing only from the description above by simply exchanging the roles of central and remote instance.

A blend of the two procedures is also possible. In such a case the central instance must be installed so that it listens to incoming connections as well as connecting to remote instances. However in every central/remote relationship only one of the pair is permitted to establish the connection!

The alarm spooler logs to the var/log/mknotifyd.log file. In the spooler configuration the loglevel can be raised so that more messages are received. With a standard loglevel one should see something like this on the central instance:

At all times the var/log/mknotifyd.state file contains the current status of the spooler and all of its connections:

A version of the same file is also present on the remote instance. There the connection will look something like this:

To test, select any monitored remote service and set it manually to CRIT with the Fake check results command.

Now on the central instance an incoming notification should appear in the notifications log file (notify.log):

The same event will look like this on the remote instance:

In the global settings, as well as the normal notifications log (notify.log) you can also alter the notification spooler’s log to a higher loglevel.

Once you have set up everything as described you will notice that on the central instance, and respectively on the remotes, a new service will be found that must definitely be taken into the monitoring. This monitors the alarmspooler and its TCP-connections. Every connection will thereby be monitored twice: once by the central, and once by the remote instance: