Monitoring Linux

Checkmk provides a number of ways to install a Linux agent — from the manual installation of its separate components, to a fully automated deployment. Some of these methods are only available in the Enterprise Editions:

When you are using the Checkmk Raw Edition you will find the Linux packages via Setup > Agents > Linux. In the Enterprise Editions clicking Setup > Agents > Windows, Linux, Solaris, AIX will lead to the Agent Bakery. From here, you find the list of agent files by clicking Related > Linux, Solaris, AIX files.

Everything required is in the first section called Packaged Agents. Finished RPM and DEB packages for the installation of a Linux agent with standard settings can be found here.

Whether you require RPM or DEB depends on the Linux distribution on which the package is to be installed:

Attention: in the Agents section you find the straight agent scripts for the manual installation to /usr/bin/. Only use these if the packaged agents do not function on your system, or if you wish to monitor Unix derivatives other than Linux.

The Checkmk Enterprise Editions utilise the Agent Bakery via a Setup module to package individually-customized agents. This process will be described in the general chapter covering the agents.

Installation of the baked packets is performed exactly as described above.

If you use the Agent Bakery, the agent’s automatic updates can set this up. This process is described in its own article.

The manual installation of an agent is rarely necessary, but is not very difficult. As well as the agent files from the site, you also require the Agents selection menu — here you will find the Checkmk Agent for Linux file:

Load this file to the target system, and copy it to a directory that is executable for root. /usr/local/bin/ is very well suited for this, since it is found in the search path and has been conceived for the user’s own extensions. Here you can also work directly with wget:

Please don’t forget the last two commands — these remove the .linux file extension and make the file executable. If everything has been done correctly the agent should now simply be an executable command which will generate its usual output. This also works if you are not in /usr/local/bin. The | head truncates everything after the 11th line:

In the case of a very old distribution (published prior to 2010) which does not recognise the timeout command, load the small program waitmax from the agent page (in the Agents box) and install it likewise in /usr/local/bin. Both timeout and waitmax perform the same function — they force a timeout when executing a program:

Waitmax was developed as a Checkmk component at a time when timeout was not widely used. It has almost the same call syntax:

Should you wish to configure or extend the agent, you will need to create the required directories yourself. The location for the three required directories is hard coded in variables that begin with MK_, and which will also be provided to the plug-ins over the environment:

These three directories should be created (with the standard permissions 755):

Should you want to change the file path, simply edit it in /usr/local/bin/check_mk_agent.

If you fundamentally want to call up the agents over SSH, a configuration for the xinetd is required, you just additionally need the SSH-configuration. How that is achieved will be described below.

The configuration per xinetd enables an access of the agent data via TCP Port 6556, and is the standard method in local networks. Install the xinetd package for this, and create the following file:

Here enter the IP address for your Checkmk server that will be permitted to access the agent under only_from. Then only an activation is required and the agent will be ready:

If you’re using systemd, you should execute the following command instead:

Once the agent has been installed, the obvious question will surely be how to test whether everything has been done correctly. All of the possibilities that are available from the Checkmk server are described in the general chapter on the agents. There are of course further diagnostic possibilities when one is directly logged into the target system itself.

Since the ‘agent’ is basically nothing more than a simple program that obtains data from your system and outputs it as loosely-formatted text, you can also invoke it as a program, and in fact do so quite easily:

Because the output can be rather long, less can also be very practical here (you can quit it with the 'Q' key):

This output does not of course prove whether the agent is also accessible over the network. But in this way it can be tested whether all of the desired data is present in the output.

Incidentally — it is not essential to be root to be able to invoke the agent, however it is possible that the output could be missing some information which requires root-permissions to obtain (e.g., multipath information and the output from ethtool).

So that possible erroneus outputs from defective plug-ins or commands do not ‘corrupt’ the real production data, as a matter of principle the agent suppresses the standard error channel. If one is looking for a specific problem this can be reactivated by invoking the agent with a special debug mode. This is done with the -d option. In so doing all of the shell commands which the agent executes will be output.

So that you can work with less here, the standard output and error channels must be combined with 2>&1:

If you have migrated your monitoring from a Nagios-based solution to Checkmk, it cannot be ruled out that you have existing check plug-ins of a standard form for which no counterpart in Checkmk (yet) exists. In most cases these will be self-written plug-ins in Perl or Shell.

The Checkmk agent offers a simple mechanism that still enables such plug-ins to be used: MK’s Remote Plugin Executor or MRPE for short. The name is intentionally an analogy to NRPE in Nagios, for which it performs the same function.

The MRPE is integral in agents and is configured with a simple text file which you yourself can save in /etc/check_mk/mrpe.cfg. In this file enter one plug-in call per line — together with the name that Checkmk should use for the service that is to be automatically created. Here is an example:

Note: The Nagios plugins may not be placed in the directory /usr/lib/check_mk_agent/plugins. This directory is reserved for the agent plug-ins. Apart from this directory, you are free to choose as long as the agent can find and run the plugins there.

If the agent is allowed to run locally, for each plug-in a new section will be found with the title <<<mrpe>>>, containing the name, exit code and output from the plug-in. This can be verified with the following practical grep-command:

The 0 or 1 in the output stand for the plug-in’s exit codes and conform to the standard pattern: 0 = OK, 1 = WARN, 2 = CRIT and 3 = UNKNOWN.

The rest will be taken care of by Checkmk automatically. Once a service discovery has been performed for the host, the two new services will be shown as available. It will look like this:

Incidentally: due to the file’s syntax the name is not permitted to contain blank characters. With the help of the same syntax as used in URLs, a space can be replaced by %20 (the ASCII-Code 32 for ‘space’ is Hexadecimal 20):

Please note that all plug-ins running in mrpe.cfg will be executed synchronously and sequentially. The plug-ins should thus not have overly long runtimes. If a plug-in hangs, all following plug-ins will be delayed. This could lead to all of the agent’s retrievals under Checkmk entering a timeout, meaning that the host can no longer be reliably monitored.

If you really require longer running plug-ins, these should be converted to asynchronous processing, thus avoiding the problem described above. For this, define a time frame in seconds for which a calculated result will be valid — for example, 300 for five minutes. To achieve this, set the expression (interval=300) following the service name in mrpe.cfg:

This will have a number of effects:

With this method tests that require a little more processing times, as well as longer intervals, can also be run without needing to make changes to the configuration on the Checkmk server.

Users of the Enterprise Editions can also configure MRPE with the Agent Bakery. The rule set Setup > Agents > Windows, Linux Solaris, AIX > Agent Rules > Generic Options > Execute MRPE checks is responsible for this. There you can configure the same things as described above. The appropriate mrpe.cfg file will then be automatically generated by the Agent Bakery.

The standard agent /usr/bin/check_mk_agent contains a whole series of sections which provide monitoring data for various checks which will then be found automatically by a service discovery. These include all of an operating system’s important monitorings.

Additionally, there is also the possibility of augmenting an agent with plug-ins. These are small scripts or programs which are invoked by an agent to include further sections with additional monitoring data. The Checkmk project delivers a whole series of such plug-ins, which — when correctly installed and configured — via a service discovery can provide new checks automatically.

Why are these plug-ins not simply built into the standard agent? For each plug-in one of the following reasons prevents such an integration:

The plug-ins for Linux and UNIX included with the project are all located on the Checkmk server in share/check_mk/agents/plugins. These are additionally available via the download page for the agents in the Setup menu (as described at the beginning of this article) in the Plugins box:

For all of our standard agent plug-ins there are also the matching check plug-ins which can evaluate the data and create services from these. They are already installed so that new services can be immediately discovered and configured.

Before installing a plug-in in an agent, please have a look at its associated file. Important tips for the correct use of the plug-in can often be found there.

The actual installation is then simple: simply save the file to /usr/lib/check_mk_agent/plugins. When doing this ensure that the file is executable. If not, use a chmod 755, otherwise the agent will not run the plug-in. Especially if the files are not transferred over scp, rather via HTTP from the download page, the execution permissions will be lost!

Once a plug-in is executable and in the correct directory, it will be invoked by the agent and a new section will be generated in the agent’s output. This section usually has the same name as the plug-in. Complex plug-ins (e.g., mk_oracle) in fact create a whole series of sections.

In older versions of the Checkmk agent the plug-in directory can be in another location. If case of uncertainty, or you don’t know if this applies in your setup, the directory can be identified with the following:

Some plug-ins require a configuration file in /etc/check_mk/ to be able to function. With others a configuration is optional and allows special features or customization. Others simply work as is. There are various sources of relevant information:

Plug-ins can process asynchronously in the same way as with MRPE. This is very useful if the plug-ins have a very long runtime, and the acquired status data in any case does not need to be refreshed every minute.

An asynchronous execution is not configured with a file. Instead, create a subdirectory in plugins whose name is a numeric, representing a count in seconds. Plug-ins in this directory will not only be executed asynchronously, at the same time by specifying the second count a minimum waiting time is also specified before the plug-in can be run again. If the agent is again called before the specified time has elapsed, it will use cached data from the last run of the plug-in. In this way an interval longer than the typical one minute can in effect be configured for the plug-in.

The following example shows how by specifying a 5 minute interval the my_foo_plugin plug-in can be changed from synchronous to asynchronous execution:

Please note that a few plug-ins are set up to execute asynchronously by default. Among these is mk_oracle. Always install such plug-ins directly in /usr/lib/check_mk_agent/plugins!

With the Checkmk Enterprise Editions included plug-ins can be configured using the Agent Bakery. This not only takes care of the installation of the plug-ins themselves, but also for the correct generation of the configuration file should one be required.

Each plug-in is configured with an agent rule. The appropriate rule set can be found in Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Agent plugins:

Since agent plug-ins are executable programs, for testing and diagnosis they can be manually started. There are however plug-ins which require specific environment variables to be set by agents — so that they can find their configuration file, for example. Set these variables manually before execution:

Some plug-ins have special invocation options for debugging. Just have a look in the plug-in!

Security is an important criterion for any software — and monitoring may be no exception. Because the monitoring agent is installed on every server being monitored, a security problem here can have serious consequences.

For this reason emphasis is placed on security in the design of Checkmk, and from the earliest days of Checkmk security has been an unshakeable principle: An agent reads no data from the network — full stop. Consequently it is quite impossible that an attacker could sneak any type of command or script element over the monitoring port 6556.

This alone provides such a high level of security that most users do without additional measures in the LAN. If the system being monitored is only accessible over an insecure internet connection, then of course quite different precedures are necessary, and an encryption with SSH would certainly be the ideal first choice.

The Checkmk agent additionally includes an inbuilt encryption, which represents a good compromise between security and complexity. In the following section we will show all of the options for protection in detail.

Even if an attacker is unable to execute commands, the monitoring data could be useful to him as among other info, the data includes a list of all processes running on the system. Is is therefore best that the data is accessible to nobody.

The best security measure when invoking a Checkmk agent is offered by invoking it via Secure Shell — in Linux in the form of an implementation of OpenSSH. This method is advisable for:

The setting-up is performed in the following steps:

And now the above procedure, step-by-step with all necessary details:

The Checkmk agent can encrypt its own data without additional tools. Strictly speaking, this is no substitute for an access control. But since an attacker cannot send commands, and can’t work with encrypted output data, as a solution this is nearly as effective.

The complexity with the use of encryption and the additional CPU-load incurred are both less than with the SSH method as described above, the method which we nevertheless still recommend when transferring data over the internet.

The encryption of course requires a suitable configuration, both on the server and in the agents. This can be created manually (Raw Edition) or with the Agent Bakery (Enterprise Editions).

To be as comprehensive as possible the monitoring of a Linux server must of course include the hardware. This is achieved in part directly with the Checkmk agents, and partly also using special plug-ins. Additionally, there are also cases in which monitoring can be implemented per SNMP, or even over a separate management board.

Modern hard drives almost always use S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology). This system continuously logs data on the condition of the HDD or SSD, and with the smart plug-in Checkmk can retrieve these values and evaluate the most important of them. In order that the plug-in functions following an installation, the following preconditions must be met:

If these prerequisites have been satisfied, and the plug-in has been installed, the data will be automatically read and appended to the agent’s output. In Checkmk the new services can then also be activated directly:

If — as shown in the screenshot — a cmd_timeout occasionally occurs, set the smart plug-in to asynchronous execution at intervals of a few minutes.

IPMI (Intelligent Platform Management Interface) is an interface for hardware management which, among other functions, enables the monitoring of hardware. Checkmk uses freeipmi for this, in order to access the hardware directly and without a network. It is installed from the package source and is ready for immediate use, so that the data will be transmitted at the next polling by Checkmk.

If freeipmi is not available, or there are other grounds preventing an installation, ipmitool can also be used. This is often already present on a system and must only served by an IPMI hardware driver — which can be provided by the openipmi package, for example. Likewise, nothing more needs to be done here subsequently. The data will be recorded by Checkmk automatically.

For error diagnosis the tools can also be executed manually in a host shell. If the freeipmi packet has been installed it can be used to control the function:

Once ipmitool has been installed the data output can be checked with the following command:

Many server producers offer their own tools for recording hardware information and delivering this via SNMP. The following prerequisites apply to be able to retrieve this data and provide it to Checkmk:

The new services for hardware monitoring thereby supported will then be automatically recognised. No further plug-ins will be required.

A management board can be configured for every host and additional data collected per SNMP. The services thereby recognised will likewise be assigned to the host.

Setting up a management board is very easy. In the host’s attributes, just enter the protocol, the IP-address and the access data for SNMP, and save the new settings:

With a service discovery the newly-discovered services will then be activated as usual.