Monitoring Linux in legacy mode

There is a comprehensive description of the installation from deb or rpm packages in the Monitoring Linux article, so we will only explain the procedure in a brief review here.

In the Checkmk Raw Edition, you can find the agent’s Linux packages via Setup > Agents > Linux. In the Checkmk Enterprise Editions, you first get to the Agent Bakery in the Setup menu via Agents > Windows, Linux, Solaris, AIX, where you will find the baked packages. From there, the Related > Linux, Solaris, AIX files menu item will take you to the list of agent files.

You can download these files via the browser or use wget or curl to download them directly into the host in the monitoring:

On RHEL, SLES and related distributions, the RPM package is installed as root with the command rpm -U:

By the way, the -U option stands for 'update' but it can also perform an initial installation correctly.

The installation of the DEB package on Debian, Ubuntu or related distributions is done as root with the command dpkg -i:

For a convenient distribution-independent installation you will need the Linux agent in the TGZ archive format ('Tarball'), which you can download from the Enterprise Editions in the Setup menu via Agents > Windows, Linux, Solaris, AIX. The TGZ archive contains the complete directory structure of the Linux agent for unpacking in the monitored host’s root directory.

The -C ('change directory') parameter is essential when unpacking to ensure that all of the file paths are correct. We also use --no-overwrite-dir so that permissions for already existing directories are not changed:

If you have done everything correctly, the agent script should now simply be executable as a command and produce its typical output. The | head truncates everything following the 11th line of output:

If a version number lower than 2.1.0 is printed here, you probably still have an older version of the agent script installed as /usr/local/bin/check_mk_agent. Move this old script, or rename it, for example by appending .bak to the filename.

A manual installation of the agent script is rarely necessary, but it is not very difficult either. In this type of installation, at first only the agent script is installed, but no configuration of the access is performed yet. For this purpose you require the Agents box from the agent files page. There you will find the file check_mk_agent.linux:

Load this file onto the target system and copy it into a directory that is executable for root. /usr/local/bin/ is a good choice, since it is in the search path and is intended for custom extensions. Alternatively, you can use /usr/bin or a subdirectory of /opt. We use /usr/bin so that all tests correspond to the other installation methods. You can also perform the installation directly with wget if available:

Do not forget the last two commands — these will remove the .linux extension and make the file executable. Now the agent should be executable as a command and produce its typical output. The | head truncates everything following the 10th line of output:

If you want to configure or extend the agent, you will need to create the necessary directories yourself. The location for the three mandatory directories is hard-coded in the agent in variables that start with MK_ and are also provided to the plug-ins via the system environment:

You should create these three directories (with the default permissions of 755 and root as owner):

If you want to use different paths, simply edit /usr/bin/check_mk_agent.

For configuring an existing xinetd that uses the /etc/xinetd.d/ directory for configuration, both the TGZ archive and the DEB and RPM packages come with a script that automates the two necessary steps: first it installs the configuration and then it makes xinetd re-read the altered settings. You have to call the script with the full file paths:

If you install the agent script manually, create the configuration file /etc/xinetd.d/check-mk-agent with the editor. The content is sufficient:

Here we have added a (commented out) line restricting access to two Checkmk servers. Further configuration options can be seen by looking at the ~/share/check_mk/agents/scripts/super-server/1_xinetd/check-mk-agent file on your Checkmk server.

If your xinetd uses the old configuration scheme with only one large /etc/xinetd.conf, transfer the sample configuration from /etc/check_mk/xinetd-service-template.cfg to /etc/xinetd.conf.

When the configuration of xinetd is complete, restart it:

You are now ready for the connection test.

First, check if your /etc/services already contains an entry for port 6556:

If this is not the case, Checkmk must be registered as a service. To do this, add the following line. The notation here is exactly the same as that stored in the IANA table with a single hyphen:

The format of the /etc/inetd.conf configuration file differs between the individual variants. Refer to the comments in the configuration file and the manual page (man 5 inetd.conf) for the format that matches your inetd. This is followed by the configuration matching openbsd-inetd with two lines for IPv4 and IPv6 support. Again, it is important to note the correct notation:

After editing the configuration file, restart inetd, e.g. with:

Depending on the init system used and the Superserver installed, this command may differ.

First check whether the xinetd or inetd could be (re)started:

Now you can connect with telnet or nc (netcat) on TCP port 6556 — first from the host itself, later from the Checkmk server:

If you receive a connection denied notification even though (x)inetd is active, check your firewall settings.

SSH works with 'public key authentication'. To do this, you first generate a pair of matched keys, where one is public and one is private. When selecting the algorithm you can choose between rsa, ecdsa or ed25519. In the example below, you use the ssh-keygen -t ed25519 command as the site user:

Leave the filename empty to use the suggested filename. Of course you can specify a different path, for example if you want to work with separate keys for individual hosts.

Important: Do not specify a passphrase! Encrypting the file with the secret key would be of no use, after all, you certainly don’t want to have to enter the passphrase every time you start the Checkmk server…​

The result is two files in the .ssh directory:

The private key is called id_ed25519 and is readable only by the site user (-rw-------) — and that’s a good thing! The public key id_ed25519.pub looks something like this:

The next step must now take place on (each of) the Linux host(s) monitored via SSH. Log in there as root and create the subdirectory .ssh in its home directory (/root), if it does not already exist. With the following command the access privileges will be set correctly to 700 right away:

Now open the authorized_keys file with a (console-based) text editor of your choice. If this file does not already exist, the editor will create it automatically:

Copy the content of the public keys into this file. This can be done e.g. with the mouse and copy & paste. Be precise! Every space counts. Also make sure that there are never two spaces in a line. And: The whole thing is a single line!. If the file already exists, simply append a new line below.

What comes now is very important! The SSH key should be used exclusively for executing the agent. SSH offers something like this under the name command restriction. To do this, put the text command="/usr/bin/check_mk_agent" at the beginning of the line you just created — separated from the rest by a single space. It will look something like this:

Save the file and check the permissions. Only the owner may have write permissions on this file.

Next, test the access to the agent with the ssh command:

The first time you will need to confirm the key’s fingerprint by entering yes. All further accesses can then be made without user interaction, including the automatic polling of the agent script by the Checkmk server every minute.

If it does not work, please check:

With very old target systems, it is also possible that keys with the elliptic curves (ed25519 and ecdsa) are unknown. In this case, generate an additional RSA key and enter this in the authorized_keys as well. SSH will then automatically use the strongest known key for the connection.

The target system has now been prepared. Now only the configuration of Checkmk itself is missing. This is done via the Setup > Agents > Other integrations> Custom integrations > Individual program call instead of agent access rule set. Create a rule here for the affected hosts and enter ssh -T root@$HOSTNAME$ or ssh -C -T root@$HOSTNAME$ (for additional compression of the agent data) as command:

You can run the connection test in the GUI under Setup > Hosts > Properties of host > Test connection to host with the Run tests button. If the test fails with timeout or access denied, check whether you used the host name in the same spelling as when testing on the command line — OpenSSH differentiates between short host name, FQDN and IP address. Alternatively you can access the host using it’s IP address. In this case you have to use the macro $HOSTADDRESS$ that is replaced by the cached (by Checkmk) IP address of the host.

After saving and executing Activate changes the host is added to the monitoring. In the monitoring the service Check-MK Agent is now displayed with the note 'Transport via SSH'. For further diagnostics the commands cmk -D and cmk -d can be used, which are explained in the article on the command line.

You can also work with more than one SSH key. Place the keys in any directory. In the Individual program call instead of agent access rule you must then specify the path to the respective private key with the -i option. It is best to use $OMD_ROOT here as a replacement for the path to the site directory (/omd/sites/mysite). The full command could then be ssh -i $OMD_ROOT/.ssh/my_key -T root@$HOSTADDRESS$, and thus the configuration would also be executable in a site with a different name:

This allows you to use different SSH keys for different groups of hosts by using multiple different rules.

To avoid providing potential attackers with plain text data despite SSH tunnels, you must disable any access to port 6556 that may still be available in monitoring on the host. If the command ss -tulpn | grep 6556 above did not find any process listening on TCP port 6556, you are done with setting up the SSH tunnel. If a line is output, the process that has been found must be permanently disabled.

The Checkmk agent script can encrypt its own data without using any additional tools. This is not a substitute for access control. However, since an attacker cannot send commands and cannot do anything with encrypted output data, the goal of security against eavesdropping is usually sufficiently fulfilled.

Of course, such encryption needs a suitable configuration on the agent as well as on the server. This can either be created manually (Raw Edition) or with the Agent Bakery (Enterprise Editions).

Note: Since symmetric encryption uses the same key for encryption and decryption, an attacker who intercepts an update packet created by the Agent Bakery which contains the key can decrypt the contents of agent output.

Even if an attacker cannot execute any commands, the agent’s monitoring data could still be useful to them, because it contains, among other things, a list of all of the processes running on the system. It is therefore best if the data cannot be easily accessed by anyone.

If you share the Checkmk Agent via xinetd, it is very easy and effective to restrict access to specific IP addresses — and those of the monitoring server, of course. This can be quickly achieved via the only_from directive in your xinetd configuration file. Enter IP addresses or address ranges (in the form 12.34.56.78/29 or 1234::/46) separated by spaces. Host names are also allowed. In this case, the system checks whether the host name determined by reverse resolution of the IP address for the requesting host matches the one entered:

In the Enterprise Editions, Agent Bakery users can configure the permitted IP addresses via the Allowed agent access via IP address (Linux, Windows) rule set. This rule set can be found via Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Generic Options.

Of course, an attacker can very easily fake their IP address and thus connect to the agent. But then it is very likely that they will not receive the response — because the response will go to the legitimate monitoring server. Or the attacker actually does get a response, but the Checkmk server doesn’t receive any data and will quickly report an error.