Distributed monitoring

Livestatus is an interface integrated into the monitoring core which enables other external programs to query status data and execute commands. Livestatus can be made available over the network so that it can be accessed by a remote Checkmk site. Checkmk’s user interface uses Livestatus to combine all tethered sites into a general overview. This then feels like a ‘large’ monitoring system.

The following diagram schematically shows the structure of a monitoring with Livestatus distributed over three locations. The Checkmk Central Site is found in the central processing site. From here central systems will be directly controlled. Additionally, there are the Remote Site 1 and Remote Site 2 which are located in other networks and controlled by their local systems:

What makes this method special is that the monitoring status of the remote sites is not sent continuously to the central site. The GUI always only retrieves data live from the remote sites when it is required by a user in the control center. The data is then compiled into a centralized view. There is thus no central data holding, which means it offers huge advantages for scaling-up!

Here are some of the advantages of this method:

In a system distributed using Livestatus as described above, it is quite possible that the individual sites can be independently maintained by different teams, and the central site only has the task of providing a centralized dashboard.

In the case of multiple, or all sites needing to be administered by the same team, a central configuration is much easier to handle. Checkmk supports this and refers to such a configuration as a distributed configuration environment. With this all hosts and services, users and permissions, time periods, and notifications, etc., will be maintained and configured on the central site, and then depending on their tasks, be automatically distributed to the remote sites.

Such a system not only has a common status overview but also a common configuration, and effectively ‘feels like a large system’.

You can even extend such a system with dedicated viewer sites, for example, that serve only as status interfaces for sub-areas or specific user groups.

No special requirements are placed on the central site. This means that a long-established site can be expanded into a distributed monitoring without requiring additional modifications.

The remote sites are then generated as new sites in the usual way with omd create. This will naturally take place on the (remote) server intended for the respective remote site.

Special notes:

Here is an example for creating a remote site with the name remote1:

The most important step is now to enable Livestatus via TCP on the network. Please note that Livestatus is not per se a secure protocol and should only be used within a secure network (secured LAN, VPN, etc.). The enabling appears per omd config as an site user on a stopped site:

Now select Distributed Monitoring:

Set LIVESTATUS_TCP to ‘on’ and enter an available port number for LIVESTATUS_TCP_PORT that is explicit on this server. The default is 6557:

After saving, start the site as normal with omd start:

Retain the password for cmkadmin. Once the remote has been subordinated to the central site, all users will likewise be replaced by those from the central site.

The remote is now ready. Verify with netstat which should show that Port 6557 is open. The connection to this port is performed by a site of the Internet superserver xinetd, which runs directly in the site:

The configuration of the distributed monitoring takes place exclusively on the central site in the menu Setup > General > Distributed monitoring, and this serves to manage the connections to the individual sites. For this function the central site itself counts as a site and is already present in the list:

Using Add connection, now define the connection to the first remote site:

In the Basic settings it is important to use the remote site’s EXACT name — as defined with omd create — as the Site ID. As always the alias can be defined as desired and also be later changed.

The settings of the Status connection determine how the central queries the status of the remote sites via Livestatus. The example in the screenshot shows a connection with the Connect via TCP(IPv4) method. This is the optimal for stable connections with short latency periods (such as, e.g. in a LAN). We will discuss the optimal settings for WAN connections later.

Enter the HTTP-URL to the remote’s web interface here (only the part preceding the check_mk/ component). If you basically access Checkmk per HTTPS, then substitute the http here with https. Further information can be found in the online help or the article on securing the web interface with HTTPS.

Replicating the configuration and thus using the distributed configuration environment is, as we discussed in the introduction, optional. Activate this if you wish to configure the remote with and from the central site. In such a case select the exact settings as shown in the image above.

A correct setting for the URL of remote site is very important. The URL must always end with /check_mk/. A connection with HTTPS is recommended, provided that the remote site’s Apache supports HTTPS. This must be installed manually on the remote at the Linux level. For the Checkmk Appliance, HTTPS can be set up using the web-based configuration interface. If you utilize a self-signed certificate, you will require the Ignore SSL certificate errors check box.

Once the mask has been saved a second site will appear in the overview:

The (so far) empty remote site’s monitoring status is now correctly integrated. In order to use the distributed configuration environment you still need a login for the remote Checkmk site. To this end, the central site exchanges a randomly-generated login secret with the remote site, through which all future communication will take place. The cmkadmin access on the remote site will subsequently no longer be used.

To login use the access data cmkadmin and the according password of the remote site. Do not forget to the box Confirm overwrite, before you click the Login button:

A successful login will be so acknowledged:

Should an error occur with the login, this could be due to a number of reasons – for example:

Points 1. and 2. can be easily tested by manually calling the remote’s URL in your browser.

When everything has been successful run Activate Changes. This will, as always, bring you to an overview of the not yet activated changes. Simultaneously the overview will also show the states of the Livestatus connections, likewise the synchronization states of the distributed configuration environment of the individual sites:

The Version column shows the Livestatus-version of the respective site. When using the CMC as the Checkmk’s core (commercial editions), the core’s version number (shown in the ‘Core’ column) is identical to that of the Livestatus. If you are using Nagios as the core (Raw Edition), the Nagios version number will be seen here.

The following symbols show replication status of the configuration environment:

In the Status column the state of the Livestatus connection for the respective site can be seen. This is shown purely for information since the configuration is not transmitted via Livestatus, but rather over HTTP. The following values are possible:

Clicking the Activate on selected sites button will now synchronize all sites and activate the changes. This is performed in parallel, so that the overall time equates to the time required by the slowest site. Included in the time is the creation of a configuration snapshot for the respective site, the transmission over HTTP, the unpacking of the snapshot on the remote site, and the activation of the changes.

Important: Do not leave the page before the synchronization has been completed on all sites — leaving the page will interrupt the synchronization.

Once your distributed environment has been installed you can begin to use it. You actually only need to tell each host by which site it should be monitored. The central site is specified by default.

The required attribute for this is ‘Monitored on site’. You can set this individually for each host. This can naturally also be performed at the folder level:

Adding hosts functions as usual — apart from the fact that the monitoring as well as the service discovery will be run from the respective remote site, there are no special considerations.

When migrating hosts from one site to another there are a couple of points to be aware of. Neither current nor historic status data from the host will be carried over. Only the host’s configuration is retained in the configuration environment. In effect it is as if the host has been removed from one site and freshly-installed on the other site:

If the continuity of the history is important to you, when implementing the monitoring you should carefully plan which host is to be monitored, and from where.

To achieve the encryption Checkmk uses the stunnel program along with its own certificate and its own Certificate Authority (CA) to sign the certificate. These will be individually generated automatically with a new site and they are therefore not predefined static CAs or certificates. That is a very important safety factor to prevent fake certificates from being used by attackers, because any attackers could then gain access to a publicly-available CA.

The generated certificates also have the following properties:

The fact that the standard certificate is valid for so long very effectively prevents you from getting connection problems that you cannot classify. At the same time it is of course possible that once a certificate has been compromised it is accordingly long open to abuse. So if you fear that an attacker will gain access to the CA or to the site certificate signed with it, always replace both certificates (CA and site)!

During an update of Checkmk, the two options LIVESTATUS_TCP and LIVESTATUS_TCP_TLS are never changed automatically. Automatic activation of TLS could ultimately result in your remote sites no longer being able to be queried.

If you have been using Livestatus unencrypted up to now and now decide to use encryption, you must activate it manually. To do this, first stop the affected sites and then activate TLS with the following command:

Since the certificates were generated automatically during the update, the site then immediately uses the new encryption feature. So that you can still access the site from the central site, please verify that the option Encryption is set to Encrypt data using TLS in the menu under Setup > General > Distributed Monitoring:

The last step is as described above — again here you first have to mark the CA of the remote site as trusted.

All accesses of a monitored host are consistently carried out from the site to which the host is assigned. This applies not only to the actual monitoring, but also to the service discovery, the Diagnostics page, the Notifications, Alert handlers and everything else. This point is very important as it is not assumed that the central site actually has access to this host.

Some of the standard views are grouped according the site from which the host will be monitored — this applies for, e.g. All hosts:

The site will likewise be shown in the host’s or service’s details:

This information is generally available for use in a column when creating your own views. There is also a filter with which a view of hosts on a specific site can be filtered:

There is a Site status snap-in for the side bar which can be added using . This displays the status of the individual sites. It also provides the option to temporarily disable and re-enable a single site by clicking on the status — or all sites at once by clicking Disable all or Enable all.

Disabled sites will be flagged with the status. With this you can also disable a site that is generating timeouts, thus avoiding superfluous timeouts. This disabling is not the same as disabling the Livestatus connection using the connection configuration in the Setup. Here the ‘disabling’ only affects the currently logged-in user and has a purely visual function. Clicking on a site’s name will display a view of all of its hosts.

In a distributed monitoring the Master control snap-in has a different appearance. Each site has its own global switch:

If you monitor with Checkmk HA-Cluster, the cluster’s individual nodes must be assigned to the same site as the cluster itself. This is because determining the clustered services’ status accesses cache files generated through monitoring the node. This data is located locally on the respective site.

Some check plug-ins use piggyback data, for example, for allocating monitoring data retrieved from an ESXi host to the individual virtual machines. For the same reason as with cluster monitoring, in distributed monitoring the ‘piggy’ (carrying) host as well as its dependent hosts must be monitored from the same site. In the case of ESXi this means that the virtual machines must be assigned to the same site in Checkmk as the ESXi system from which the monitoring data is collected. This can mean that it is better to poll the ESXi host system directly rather than to poll a global vCenter. Details for this can be found in the documentation on ESXi monitoring.

The Checkmk Hardware/Software inventory also functions in distributed environments. In doing so the inventory data from the var/check_mk/inventory directory must be regularly transmitted from the remotes to the central site. For performance reasons the user interface always accesses this directory locally.

In the commercial editions the synchronization is carried out automatically on all sites that are connected using the Livestatus proxy.

If you run inventories using the Raw Edition in distributed systems, the directory must be regularly mirrored to the central site with your own tools (e.g., with rsync).

Even when all sites are being centrally monitored, a login on an individual site’s interface is quite possible and often also appropriate. For this reason Checkmk ensures that a user’s password is always the same for all sites.

A password change made by the administrator will take effect automatically as soon as it is shared to all sites with Activate Changes.

A change made by a user themselves using the sidebar in their personal settings works somewhat differently. This cannot execute an Activate changes since the user of course has no general authority for this function. In such a case Checkmk will automatically share the changed password across all sites — directly after it has been saved in fact.

As we all know, networks are never 100 % available. If a site is unreachable at the time of a password change, it will not receive the new password. Until the administrator successfully runs an Activate changes, or respectively, the next successful password change, this site will retain the old password for the user. A status symbol will inform the user of the status of the password sharing to the individual sites.

It is self-evident that the host tags used in the remote must also be known to the central site in order that they can be carried over. Check these before the migration and add any missing tags to the central site manually. Here it is essential that the Tag-IDs match — the tag’s title is irrelevant.

Next, move the hosts and rules into the configuration environment on the central site. This only works for hosts and rules in sub-folders (i.e., not in the Main folder). Hosts in the main folder should first simply be moved into a remote site’s sub-folder via Setup > Hosts > Hosts.

The actual migration can then be achieved quite simply by copying the appropriate directories. Each folder in configuration environment corresponds to a directory within ~/etc/check_mk/conf.d/wato/. These can be copied using a tool of your choice (e.g. scp) from the tethered site to the same location in the central site. If a directory with the same name already exists there, simply rename it. Please note that Linux users and groups are also used by the central site.

Following the copying the hosts will appear in the central site’s Setup — as well as the rules you have created in these folders. The folders’ properties will also be included with the copying. These can be found in the directory in the hidden .wato file.

So that the attributes of the central site’s parent folder’s functions are correctly inherited, as a final step following the migration the parent folders’ characteristics must be opened and saved once – the host’s attributes will thereby be freshly defined.

Installing the Livestatus proxy is very simple. It is activated by default in the commercial editions — which can be seen when starting a site:

Select the setting ‘Use Livestatus Proxy-Daemon’ for the connection to the remote sites instead of ‘Connect via TCP’:

The details for host and port are as always. No changes must be made on the remote site. In Number of channels to keep open enter the number of parallel TCP-connections the proxy should establish and sustain to the target site.

The TCP-connections pool is shared by all GUI inquiries. The number of connections limits the maximum number of queries that can be processed concurrently. This indirectly limits the number of users. In situations in which all channels are reserved this will not immediately lead to an error. The GUI waits a given time for a free channel. Most queries actually require only a few milliseconds.

If the GUI must wait longer than Timeout waiting for a free channel for a channel, it will be interrupted with an error and the user will receive an error message. In such a case the number of connections should be increased. Be aware however that on the remote site sufficient parallel incoming connections must be allowed — this is set to 20 by default. This setting can be found in the global options under Monitoring core > Maximum concurrent Livestatus connections.

The Regular heartbeat provides a constantly active monitoring of the connections directly at the protocol level. In the process the proxy regularly sends a simple Livestatus query which must be answered by the remote site within the predetermined time (default: 2 seconds). With this method a situation where the target server and the TCP-port are actually reachable, but the monitoring core no longer responds, will also be detected.

If a response fails to appear, all connections will be declared ‘dead’, and following a ‘cooldown’ time (default: 4 seconds) will be newly established. All this takes place proactively — i.e. without a user needing to open a GUI-window. In this way outages can be quickly detected, and via a recovery the connections can be immediately reestablished and in the best case be available before a user even notices the outage.

The Caching ensures that static queries need only be responded-to once by the remote site, and from that point of time can be responded to directly and locally, without delay. An example of this is the list of monitored hosts required by Quicksearch.

The Livestatus proxy has its own log file which can be found under var/log/liveproxyd.log. On a correctly-configured remote site with five channels (standard) it will look something like this:

The Livestatus proxy regularly records its state in the var/log/liveproxyd.state file:

And when a site is currently stopped the state will look like this:

Here the state is ‘starting’. The proxy is thus attempting to establish connections. There are no channels yet. During this state queries to the site will be answered with an error.

First, on the remote site, create a copy of its host’s and service’s configurations in Nagios-configuration format. Also redirect the output from livedump -TC to a file:

The start of the file will look something like this:

Transmit the file to the central site, (e.g. with scp) and save them there in the ~/etc/nagios/conf.d/ directory – here Nagios expects to find the configuration data for hosts and services. Select a file name that ends with .cfg, for example ~/etc/nagios/conf.d/config-remote1.cfg. If an SSH-access from remote to central site is possible it can be done, for example, as below:

Now log in to the central site and activate the changes:

Now all of the remote’s hosts and services should appear in the central site – initially with the PEND state, which they will retain for the time being:

Note:

Once the hosts are visible in the central site, we will need to setup a (regular) transmission of the remotes' monitoring status. Again create a file with livedump, but this time without secondary options:

This file contains the states of all hosts and services in a format which Nagios can read directly from check results. The start of this file looks something like this:

Copy this file to the central site into the ~/tmp/nagios/checkresults directory. Important: This file’s name must begin with c and be seven characters long. With scp it will look something like this:

Finally, create an empty file on the central site with the same name and the .ok extension. With this Nagios will know that the status file has been transferred completely and can now be read in:

The status of the remotes’ hosts/services will now be immediately updated on the central site:

The transmission of the status must from now on be made regularly. Livedump unfortunately doesn’t support this task and you will need to script it yourself. The livedump-ssh-recv script can be found in ~/share/check_mk/doc/treasures/livedump, which you can employ in order to receive Livedump updates (including those from the configuration) on the central site per SSH. For details, see the comments in the script itself.

The configuration and status dump can also be restricted by using Livestatus filters. For example, you could limit the hosts to the members of the mygroup host group:

Further information on Livedump can be found in the README file in the directory ~/share/doc/check_mk/treasures/livedump.

The commercial editions provide a built-in mechanism for centralized notifications which can be individually activated for each remote site. Such remotes then route all notifications to the central site for further processing. The centralized notification is thereby independent of whether the distributed monitoring has been set up in the standard way, or with CMCDump, or by using a blend of these procedures. Technically speaking, the central notification server does not even need to be the ‘central’. This task can be taken on by any Checkmk site.

If a remote site has been set to ‘forwarding’, all notifications will be forwarded directly to the central site as they would be from the core — effectively in a raw format. Once there the notification rules will be evaluated which actually decide who should be notified and how. The required notification scripts will be invoked on the central site.

The notification spoolers of remote and central (notification) sites communicate with each other via TCP. Notifications are sent from remote to central site. The central acknowledges to the remotes that the notifications have been received, which prevents notifications being lost even if the TCP connection is broken.

There are two alternatives for the construction of a TCP-connection:

Consequently there is nothing standing in the way of forwarding notifications if for network reasons establishing connections is only possible in a specific direction. The TCP-connections are supervised by the spooler with a heartbeat signal and are immediately reestablished as needed — not only in the event of a notification.

Since remote and central site require different global settings you must make site specific settings for all remotes. Configuring the central site is performed using the normal global settings. Please note — these settings will be automatically inherited by all remotes for which no specific settings have been defined.

Let us look first at an example where the central site establishes the TCP-connections to the remote sites.

Step 1: On the remote site, edit the site specific global setting Notifications > Notification Spooler Configuration and activate Accept incoming TCP connections. TCP-Port 6555 will be recommended for incoming connections. If there are no objections, adopt these settings.

Step 2: Now, likewise, in the Notification Spooling submenu only on the remote site, select the option Forward to remote site by notification spooler.

Step 3: Now, on the central site — i.e. in the normal global settings — configure the connection to the remote site (and then to additional remotes as needed):

Step 4: Set the global setting Notification Spooling to Asynchronous local delivery by notification spooler, so that the central site’s communications will also be processed over the same central spooler.

Step 5: Activate the changes.

If the TCP-connection should be established from the remote outwards, the procedure is identical, differing only from the description above by simply exchanging the roles of central and remote site.

A blend of the two procedures is also possible. In such a case the central site must be installed so that it listens to incoming connections as well as connecting to remote sites. However in every central/remote relationship only one of the pair is permitted to establish the connection!

The notification spooler logs to the var/log/mknotifyd.log file. In the spooler configuration the log level can be raised so that more messages are received. With a standard log level one should see something like this on the central site:

At all times the var/log/mknotifyd.state file contains the current status of the spooler and all of its connections:

A version of the same file is also present on the remote site. There the connection will look something like this:

To test, select any monitored remote service and set it manually to CRIT with the Fake check results command.

Now on the central site an incoming notification should appear in the notifications log file (notify.log):

The same event will look like this on the remote site:

In the global settings you can change both the normal log file for notifications (notify.log) and the log file of the notification spooler to a higher log level.

Once you have set up everything as described you will notice that on the central site, and respectively on the remotes, a new service will be found that must definitely be taken into the monitoring. This monitors the notification spooler and its TCP-connections. Every connection will thereby be monitored twice: once by the central, and once by the remote site: