Monitoring Linux

You install the Linux agent by installing the RPM or the DEB package. Whether you need RPM or DEB depends on the Linux distribution on which you want to install the package:

Before installation you will need to get the package and bring it to the host (for example with scp or WinSCP) where you want the agent to run.

After you have fetched the RPM or the DEB package and — if necessary — copied it to the host to be monitored using scp, WinSCP or other means, the installation is accomplished with a single command.

The RPM package is installed under root with the command rpm -U:

By the way, the -U option stands for 'update', but it can also perform an initial installation correctly. This also means that you can use this command to update an existing agent to the current version — and also use the same command for future updates of the agent package.

The installation of the DEB package — and an update — is done under root with the command dpkg -i:

Here the package was installed for the first time on a previously agentless host. The cmk-agent user has been created and systemd has been configured. We address the interim warning about insecure mode, i.e. legacy pull mode, in a moment.

The Checkmk Enterprise Editions have a software module, the Agent Bakery, for automatically packaging customized agents. A detailed description of this can be found in the general article on the agents. Installation of the baked packages is done in the same way as described above for the included packages.

If you use the Agent Bakery, you can also set up automatic updates of the agent. These updates are described in their own article.

If the Agent Controller could be configured with systemd during installation, the next step is the registration, which sets up TLS encryption so that the encrypted agent output can be decrypted by the Checkmk server and then displayed in the monitoring.

There is a special feature available when the agent is installed with Agent Controller for the first time. In that case, the agent switches to the unencrypted legacy pull mode so that the Checkmk server is not cut off from the monitoring data and can continue to display it. This affects both a new installation and an update of an agent of version 2.0.0 and older.

You will receive a notice of the activated legacy pull mode in the command output during the installation of the agent. It will look something like this in the monitoring:

The Checkmk site recognizes from the agent output that the Agent Controller is present and thus TLS encryption is possible — but has not yet been enabled. The Check_MK Agent service changes to the WARN state and remains so until you register it. After registration, only the encrypted pull mode will be used for communication. The legacy pull mode is switched off and will remain so. However, it can be switched on again by command if necessary.

The situation is different if the Agent Controller could not be registered as a daemon with systemd during the installation. In such a case, host registration is not possible and the only available communication path remains the legacy mode. In the next chapter, you can determine whether you can proceed with registration by testing the Agent Controller and system environment.

Note: In the Checkmk Agent installation auditing rule set you will find various settings to check the state of the agent and make it visible in monitoring. Among other things, you can specify here which state the Check_MK Agent service should have if TLS configuration has not yet been performed.

Immediately after the installation of the new agent (also as an update of an agent of version 2.0.0 and older) only unencrypted communication in legacy pull mode is possible. An exclusively encrypted data transmission is only active after a trust relationship has been established.

Therefore, perform the registration promptly after installation/update. This chapter shows how to do this.

Registration, and thus the establishment of the mutual trust relationship, is done from a Checkmk user with access to the REST API. For this purpose, a good choice is the automation user, which is automatically created with every Checkmk installation and whose password you can randomly generate.

First create the new host via Setup > Hosts > Add host. A host must of course exist in the configuration environment before it can be registered.

The agent with the Agent Controller requires a Linux distribution with systemd, more precisely systemd in a version 220 or newer.

There is a good chance that this requirement is met on your host, since from 2015 most Linux distributions have adopted systemd as their init system, replacing other init systems such as SysVinit, e.g. SUSE Linux Enterprise Server from version 12, openSUSE from version 12.1, Red Hat Enterprise Linux from version 7, Fedora from version 15, Debian from version 8 and Ubuntu from version 15.04. Unfortunately, comparing the version number alone does not bring certainty, since systemd may be missing even on a current Linux system if it has 'only' been updated over the years.

Therefore, check on the host on which the agent is to be installed whether systemd is running and in which version:

The above command output shows that systemd is installed in the correct version. If systemd is not running, or is running in a version that is too old, the Agent Controller cannot be used. Complete the setup as described in the article Monitoring Linux in legacy mode.

Now check whether the Agent Controller can be started:

The version number should be shown in the output, for example:

In rare cases, the following error message may appear:

The reason for this is that your Linux uses a different computer architecture than x86_64, for example the older 32-bit x86 or ARM. In this case, the Agent Controller cannot be used. Complete the setup as described in the article Monitoring Linux in legacy mode.

The next step is to find out which program is waiting for requests on port 6556:

Here it is cmk-agent-ctl, thus the requirements for an encrypted communication have been fulfilled. If however systemd, xinetd or inetd are within the parentheses the prerequisites for using the Agent Controller are not met. In such a case, also complete the setup as described in the article Monitor Linux in legacy mode.

Registration is made using the agent cmk-agent-ctl controller, which provides a command interface for configuring the connections. You can use the cmk-agent-ctl help command to display help on the options.

Next, go to the host that is to be registered. Here, with root privileges, make a request to the Checkmk site:

The host name following the --hostname option must be exactly the same as it was when it was created in the Setup. The --server and --site options specify the name of the Checkmk server and the site. The server name may also be the IP address, the site name (here mysite) corresponds to the one you see in the URL path for the web interface. The options are completed by the name and password used by the automation user. If you omit the --password option, the password will be requested interactively.

If the specified values were correct, you will be asked to confirm the identity of the Checkmk site to which you want to connect. For clarity here, we have abbreviated the server certificate to be confirmed:

Confirm with Y to complete the process.

If no error message is displayed, the encrypted connection will have been established. All data will now be transmitted in compressed form via this connection.

The cmk-agent-ctl status command should now show precisely one trust relationship with the Checkmk server:

Note: There can only ever be one trust relationship between host and site. For example, if you register the already registered host mynewhost under a different name (mynewhost2) but with the same IP address, then the new connection will replace the existing one. The connection from mynewhost to the site will be disconnected and no more agent data will be supplied to the monitoring from that host.

For easier registration of multiple hosts, any host on which the agent is installed can perform a registration on behalf of other hosts. The registration process exports a JSON file, which can then be transferred to the target host and imported there. Again, as before, the host registered in the job must already be set up on the site.

First, on any host in the Setup, the registration is performed by proxy. Here, of course, the Checkmk server comes in handy, as it is usually the first host to be set up. As with the example above, you can pass the password by option or be asked for it interactively if you omit the --password option. We redirect the JSON output to a file in the example:

Next we transfer the /tmp/mynewhost3.json file to the host we registered for and import that file:

This process is also possible in a single step using a pipeline where the output of cmk-agent-ctl proxy-register is handed over as input to ssh hostname cmk-agent-ctl import:

Once the registration is complete, perform an connection test and a service discovery in the Checkmk server Setup. Then, as a final step, add the discovered services to the monitoring by activating the changes.

If the connection test fails, refer to the following chapter for information on testing and error diagnosis.

You can also deregister a host.

On a host connected to the Checkmk server, you can revoke the trust. Here, in the following command, the Universally Unique Identifier (UUID) to specify is the one output by the cmk-agent-ctl status command:

To delete all connections from the host and additionally restore legacy pull mode, enter the following command:

After that, the agent behaves as it did after the initial installation and before the first registration and sends its data unencrypted.

Complete the deregistration on the Checkmk server: In the Setup, on the Properties of host page, select the Host > Remove TLS registration menu item and confirm the prompt.

In case you prefer the command line: On the Checkmk server, for each connection of a host that is in monitoring, there is a soft link with the UUID that points to the folder with the agent output:

If you delete this soft link, you will need to re-register the host.

The agent script is a simple shell script that obtains data on your system and outputs it as loosely formatted text. You can call this script directly from the command line. Since the output can be a bit long, the option less to scroll the output is very handy here. You can exit it with the Q key:

This is how you determine whether all of the desired data is included in the output — for example, whether all of the installed plug-ins are providing their expected output.

By the way, you do not have to be root to call the agent. However, the output will then lack some information that requires root privileges to obtain (e.g. multipath information and the outputs of ethtool).

To prevent any error output from inactive plug-ins or commands from 'contaminating' the required data, the agent script generally suppresses the standard error channel (STDERR). If you are looking for a specific problem, you can re-enable the STDERR by calling the agent script in a special debug mode. You do this with the -d option. This will also print all shell commands that the script executes.

To be able to work with less here, you have to combine standard output (STDOUT) and error channel with 2>&1:

If registering a host fails even before a certificate is presented, knowledge about the ways of communication can help identifying the problem — and of course solving it.

After entering the cmk-agent-ctl register command, the Agent Controller first asks the Checkmk server for the Agent Receiver port using the REST API. As second step a connection to the Agent Receiver is established to request the certificate. You can simulate the first request on the host with a program like curl:

The parameter --insecure instructs curl to skip the certificate check. This behavior reflects the behavior of the Agent Controller in this step. The response is only a few bytes, containing the port number of the Agent Receiver. For the first site this is usually just 8000, for the second 8001 and so on. Common problems regarding this request are:

When the curl command fails you might change routing or firewall settings to enable access, respectively add the internal CA to the trust chain of the host.

In case the host you are trying to register uses an HTTP proxy, curl will use it, but cmk-agent-ctl won’t do so with default settings. Use the additional --detect-proxy option to instruct cmk-agent-ctl to use a proxy configured via system settings.

However often it may be easier to find out the port of the Agent Receiver and note it down. To do so, on the Checkmk server run, logged in as site user:

Now you can specify the port when entering the command for registration. This skips the first request to the REST API. Communication then takes place directly with the Agent Receiver without any detours:

Port 8000 also must be reachable from the host. In case it is not, you will get this error message:

Equivalent to port 443 (respectively 80) mentioned above, you can now adjust routing or firewall settings so that the host to be registered can reach the Checkmk server on the Agent Receiver’s port (8000 or 8001…​)

Should security policies prohibit access to the Agent Receiver, there is still the possibility to use registration by proxy on the Checkmk server.

The Agent Controller provides its own dump subcommand that displays the full agent output as it arrives in the monitoring:

This allows you to verify that the data from the agent script has arrived at the Agent Controller. This output does not yet prove that the agent is also reachable over the network however.

In some cases, the output will look like this:

This would be the case when the Agent Controller daemon is not running in the background — immediately following an update, for example. Restart this background process:

If cmk-agent-ctl dump fails again, check if and which program is listening on port 6556:

If the output is empty or there is a command other than cmk-agent-ctl within the parentheses, the system requirements for using the Agent Controller have not been met. In this case, complete the setup as described in the Monitor Linux in legacy mode article.

If it has been verified that the agent script and its installed plug-ins are executing as expected locally, you can next check via netcat (or nc) whether port 6556 is reachable via the external IP address of the host:

The output 16 indicates that the connection was successfully established and that the TLS handshake can now take place. Since everything else here is TLS encrypted, no more detailed check is possible.

If a remote connection test fails, it is usually due to the firewall setting. In this case, configure iptables or nftables to allow access to TCP port 6556 from the Checkmk server.

If the communication between agent and Checkmk server is still unencrypted (as in legacy pull mode), or is and will remain unencrypted (as in legacy mode), this command will give you the full unencrypted agent output instead of the 16.

Note: For more diagnostics to run on the Checkmk server, see the general article on monitoring agents.

If the output remains unencrypted even after trying to register, use grep to determine the status from the output:

If the allow_legacy_pull variable is set to false, the Agent Controller itself does not allow plaintext output, but another service, for example xinetd, is responsible for TCP port 6556. This a condition occasionally encountered after updating a system that does not meet the requirements for using the Agent Controller. In this case, first perform a deregistration and then complete the setup as described in the Monitor Linux in legacy mode article.

Security is an important criterion for any software, and monitoring is no exception. Since the monitoring agent is installed on every monitored server, a security problem here would have particularly serious consequences.

This is why security was emphasized in the design of Checkmk and has been an absolute principle since the earliest days of Checkmk: The agent does not read data from the network. Period. This means that it is impossible for an attacker to inject any commands or script components via the monitoring port 6556.

For an attacker, however, even a process list can be a first approach for drawing conclusions about worthwhile targets. Therefore, transport encryption between agent and Checkmk server with Transport Layer Security (TLS) is mandatory from Checkmk version 2.1.0. Here, the Checkmk server 'pings' the monitored host, which then establishes the TLS connection to the Checkmk server and transmits the agent output over it. Since only Checkmk servers with which a trust relationship exists can initiate this data transfer, there is no risk of data falling into the wrong hands.

Since only authorized Checkmk servers can retrieve data and unauthorized servers fail after a few bytes of handshake, the risk of a Denial of Service (DoS) attack is very low. For this reason, no further access restriction is currently planned. Of course you can block port 6556 against unauthorized access via iptables. Any rule that may exist and which has been transferred to clients via the Agent Bakery to restrict access to certain IP addresses is ignored by the Agent Controller.

Especially when updating the agent, it may be that the built-in (symmetric) encryption is active, which is performed by the agent script itself. If TLS encryption and built-in encryption are active at the same time, then the entropy of the transmitted data is so high that compression, which is active from version 2.1.0 onwards, will not save any transmitted data — and will burden the CPUs of both the host and the Checkmk server with additional further encryption and decryption steps.

For this reason, you should disable the built-in encryption promptly after switching to TLS. In the Checkmk Raw Edition you can do this by renaming the /etc/check_mk/encryption.cfg configuration file.

In the Checkmk Enterprise Editions you can change existing rules to Use TLS encryption in Setup > Agents > Access to agents > Encryption (Linux, Windows) in the Encryption (Linux, Windows) section and then rebake the agent packages. After the next automatic agent update, the agent script encryption is disabled but guaranteed by the Agent Controller. Note that after the automatic agent update, only registered hosts can provide monitoring data.

The /usr/bin/check_mk_agent agent script contains a whole set of sections which provide monitoring data for various check plug-ins which are then automatically found by the service detection. This includes all of the important monitoring for the operating system.

In addition, there is the possibility of extending the agent with agent plug-ins. These are small scripts or programs that are called by the agent and extend it with additional sections containing additional monitoring data. The Checkmk project delivers a whole series of such plug-ins, which — if they are correctly installed and configured — automatically deliver new services through a service detection.

Why aren’t these plug-ins simply hard-coded into the agent? For each of the plug-ins there will be one of the following reasons:

The plug-ins included with Linux and Unix can all be found on the Checkmk server under share/check_mk/agents/plugins. They are also available from the agents download page in the Setup menu (as described in the installation chapter) in the Plugins box:

For all of the agent plug-ins we provide, there are matching check plug-ins that can evaluate the agent’s data and create services. These are already installed, so that newly found services can be detected and configured immediately.

Note: Before you install a plug-in on a host, take a look at its corresponding file. Often you will find important information there about the correct use of the plug-in.

The actual installation is then simple: Copy the file to /usr/lib/check_mk_agent/plugins. Make sure that it is executable. If not, use a chmod 755, otherwise the agent will not execute the plug-in. Note that especially if you do not transfer the files via scp but fetch them via HTTP from the download page, the execution permission will be lost.

Once the plug-in is executable and located in the correct directory, it will be automatically invoked by the agent and a new section will be created in the agent output. This section usually has the same name as the plug-in. Complex plug-ins, such as mk_oracle for example, even create a whole series of new sections.

Some plug-ins will require a configuration file in /etc/check_mk/ in order to work. For others, a configuration is optional and enables special features or customizations. Still others will work simply as they are. There are several sources of information on a plug-in:

Just as with MRPE, you can also run plug-ins asynchronously. This is very useful if the plug-ins have a long runtime and the obtained status data does not need to be regenerated every minute anyway.

Asynchronous execution is not configured via a file, instead you create a subdirectory under /usr/lib/check_mk_agent/plugins whose name is a number: a number of seconds. Plug-ins in this directory are not only executed asynchronously, but at the same time you specify a minimum waiting time with the number of seconds before the plug-in should be executed again. If the agent is queried again before the time expires, it uses cached data from the last time the plug-in was executed. This allows you to configure a longer interval for the plug-in than the typical one minute.

The following example shows how to change the my_foo_plugin plug-in from synchronous execution to asynchronous execution with an interval of 5 minutes (or 300 seconds):

Note: Some plug-ins automatically implement asynchronous execution. This includes mk_oracle. Install such plug-ins directly after /usr/lib/check_mk_agent/plugins.

In the Checkmk Enterprise Editions, the included plug-ins can be configured via the Agent Bakery. This takes care of both the installation of the actual plug-in and the correct creation of its configuration file, should one be needed.

Each plug-in is configured via an agent rule. You can find the appropriate rule sets in Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Agent Plugins:

Since agent plug-ins are executable programs, you can also run them manually for testing and diagnostic purposes. However, there are plug-ins that need certain environment variables set by the agent to find their configuration file, for example. Set these variables manually before execution:

Some plug-ins also use special call options for debugging. Simply take a look at the plug-in file.

There are two good reasons to continue using Nagios plug-ins under Checkmk. If you have migrated your monitoring from a Nagios based solution to Checkmk, you can continue to use older check plug-ins for which there is no Checkmk equivalent yet. In many cases these are self-written plug-ins in Perl or shell.

The second reason for using Nagios plug-ins is true end-to-end monitoring. Let’s assume you have your Checkmk server, a web server and a database server distributed over a large data center. In such a case, the database server response times measured from the Checkmk server are not very meaningful. It is far more important to know these values for the connection between the web server and the database server.

The Checkmk agent provides a simple mechanism to meet these two requirements: MK’s Remote Plugin Executor or MRPE for short. The name is deliberately an analogy to the NRPE of Nagios, which performs the same task there.

The MRPE is built into the agent and is configured with a simple text file, which you create as /etc/check_mk/mrpe.cfg. There you specify one plug-in call per line — along with the name you want Checkmk to use for the service it automatically creates for it. Here is an example:

Note: The Nagios plugins may not be placed in the directory /usr/lib/check_mk_agent/plugins. This directory is reserved for the agent plug-ins. Apart from this directory, you are free to choose as long as the agent can find and run the plugins there.

If you now run the agent locally, you will find a new section for each plug-in called <<mrpe>> which contains the name, exit code and output from the plug-in. You can check this with the following handy grep command:

The 0 and 1 in the output stand for the exit codes of the plug-ins and follow the conventional scheme: 0 = OK, 1 = WARN, 2 = CRIT and 3 = UNKNOWN.

The rest will now be done automatically by Checkmk. Once you invoke a service discovery for the host, the two new services will show up as available. It will look like this:

By the way, due to the syntax of the file, the name cannot contain spaces. However, you can replace a space with %20 using the same syntax as in URLs (ASCII code 32 for space is hexadecimal 20):

Note that all plug-ins you list in mrpe.cfg will be executed synchronously in order. The plug-ins should therefore not have too long an execution time. If one plug-in hangs, the execution of all others will be delayed. This can lead to the complete querying of the agent by Checkmk running into a timeout and preventing the host from being reliably monitored.

If you really need longer running plug-ins, you should switch them to asynchronous execution and thus avoid such problems. To do this, set a time in seconds during which a calculated result should remain valid, e.g. 300 for five minutes. To do this, set the expression (interval=300) following the service name in mrpe.cfg:

This facility has several benefits:

So this allows you to run tests that need a bit more computing time as well as over longer intervals, without having to configure anything on the Checkmk server.

Users of Enterprise Editions can also configure MRPE with the Agent Bakery. Responsible for this is the rule set Setup > Agents > Windows, Linux Solaris, AIX > Agent Rules > Generic Options > Execute MRPE checks. There you can configure the same things as described above. The file mrpe.cfg will then be generated automatically by the bakery.

Modern hard drives almost always have S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology). This system continuously records data on the state of the HDD or SSD and Checkmk can retrieve these values with the smart plug-in and evaluate the most important of them. For the plug-in to work after installation, the following requirements must be met:

If these requirements are met and the plug-in is installed, the data is automatically read and appended to the agent’s output. In Checkmk you can then also activate the new services directly:

As seen in the screenshot, if cmd_timeout occasionally occurs, switch the plug-in to asynchronous execution at intervals of a few minutes.

IPMI (Intelligent Platform Management Interface) is an interface for hardware management which also enables monitoring of the hardware. Checkmk uses freeipmi for this purpose to access the hardware directly and without a network. freeipmi is installed from the package sources and is then ready for immediate use, so that the data will be transmitted the very next time Checkmk is called.

If freeipmi is not available or there are other reasons not to install it, ipmitool can also be used. ipmitool is often already present on the system and only needs to be supplied with an IPMI device driver, such as that provided by the openipmi package. Again, you do not need to do anything else after an installation, and the data will be collected automatically by Checkmk.

For error diagnosis you can also run the tools manually in a host shell. Once you have installed the freeipmi package, you can check its functions with this:

If ipmitool has been installed, you can check the output of its data with the following command:

Many large server manufacturers also provide their own tools for collecting the hardware information and making it available via SNMP. The following prerequisites must be met in order to retrieve this data and provide it to Checkmk:

The new services for hardware monitoring supported by this are then automatically detected and no further plug-ins are required.

A management board can be configured for each host and additional data can be retrieved via SNMP. The services detected in this way are then also assigned to the host.

Setting up the management board is very simple. Simply enter the protocol, the IP address and the access data for SNMP in the host’s properties and save these new settings:

Configure the management board for SNMP in the properties of the host in the Setup.

With a service discovery, the newly discovered services will then be then enabled as usual.