Monitoring Windows

You install the Windows agent by installing the MSI package.

Before installation, you need to get the package and bring it to the host (for example with scp or WinSCP) on which the agent will run.

The Checkmk Enterprise Editions have the Agent Bakery, a software module for automatically packaging customized agents — including Windows agents. For a detailed description of this, see the general article on agents. The installation of the baked MSI package is then done again exactly as described above.

If you use the Agent Bakery, you can set up automatic updates of the agent. These are described in their own article.

During an installation the MSI package stores the program-specific files in C:\Program Files (x86)\checkmk\service\ and the host-specific files in C:\ProgramData\checkmk\agent\. You do not need to customize the program-specific files. The host-specific files are used to store plug-ins, log and configuration files and to configure the behavior of the agent.

Note: By default, the entire C:\ProgramData directory is hidden in Windows.

The agent reads three configuration files in succession:

If an option has been set in multiple files, then the last file read in determines the content of this option. For manual work with the agent, therefore, only the last configuration file check_mk.user.yaml is relevant, because it is read in last and thus has the last word. If the agent bakery is not used, this is in fact the only file in which customizations to the agent’s configuration may be made.

As you may have already recognized from the file extension of the configuration files, YAML is used as the file format.

After installing the agent including the Agent Controller, the next step is the registration, which sets up TLS encryption so that the encrypted agent output can be decrypted by the Checkmk server and displayed in monitoring.

There is a special feature when the agent was installed with the Agent Controller for the first time. Then the agent switches to the unencrypted legacy pull mode so that the Checkmk server is not cut off from the monitoring data and can continue to display it. This applies to a new installation as well as to an update of an agent of version 2.0.0 and older.

It will look like this in the monitoring:

The Checkmk site recognizes from the agent output that the Agent Controller is present and thus TLS encryption is possible — but not yet enabled. The Check_MK Agent service changes to the WARN state and remains so until you register it. After registration, only encrypted pull mode is used for communication. The legacy pull mode is switched off and will remain so. However, it can be switched on again by command if necessary.

The case is different if the agent is installed on very old Windows systems. The installation of the agent may be successful. However, if the Agent Controller cannot be started due to API incompatibilities, the unencrypted transfer is resorted to. Without Agent Controller, registration is not possible and the only communication path remains the legacy mode. In legacy mode, only the sections in the Registration chapter are relevant for adding the host to the Setup and then to the monitoring. In the chapter Testing and troubleshooting you must omit the test for calling the Agent Controller, because it is not available in legacy mode. Since there is also no TLS encryption without Agent Controller, you have to use other methods of encryption if required. We recommend using the built-in (symmetric) encryption in this case with the Encryption (Linux, Windows) rule and/or setting up an SSH tunnel.

Note: In the Checkmk Agent installation auditing rule set you will find various settings to check the state of the agent and make it visible in monitoring. Among other things, you can specify here which state the Check_MK Agent service should have if TLS configuration has not yet been performed.

Immediately after the installation of the new agent (also as an update of an agent of version 2.0.0 and older) only unencrypted communication in legacy pull mode is possible. An exclusively encrypted data transmission is only active after a trust relationship has been established.

Therefore, perform the registration promptly after installation/update. This chapter shows how to do this.

Registration, and thus the establishment of the mutual trust relationship, is done from a Checkmk user with access to the REST API. For this purpose, a good choice is the automation user, which is automatically created with every Checkmk installation and whose password you can randomly generate.

Note: Since there is no Agent Controller, and thus no registry and TLS encryption, on very old Windows systems, you have to use other methods of encryption if required. In this case we recommend using the built-in (symmetric) encryption with the rule Encryption (Linux, Windows) or to set up an SSH tunnel.

First create the new host via Setup > Hosts > Add host. A host must exist in the configuration environment before it can be registered.

The registration is done using the Agent Controller cmk-agent-ctl, which provides a command interface for configuring the connections. You can use the cmk-agent-ctl help command to display help on the options.

Now go to the host that is to be registered. Here you have to make a request to the Checkmk site with administrator rights:

The host name following the --hostname option must be exactly the same as it was when it was created in the Setup. The --server and --site options specify the name of the Checkmk server and the site. The server name may also be the IP address, the site name (here mysite) corresponds to the one you see in the URL path of the web interface. The options are completed by the name and password for the automation user. If you omit the --password option, the password will be requested interactively.

If the specified values were correct, you will be asked to confirm the identity of the Checkmk site to which you want to connect. We have shortened the server certificate to be confirmed for clarity:

Confirm with Y to complete the process.

If no error message is displayed, the encrypted connection has been established. All data will be transmitted in compressed form via this connection.

The cmk-agent-ctl status command now shows exactly one trust relationship with the Checkmk server:

Note: There can only ever be one trust relationship between host and site. For example, if you register an already registered host mynewhost under a different name (mynewhost2) but with the same IP address, then the new connection will replace the existing one. The connection from mynewhost to the site will be disconnected and no more agent data will be supplied to the host for monitoring.

For easier registration of multiple hosts, any host on which the agent is installed can perform registration on behalf of others. This exports a JSON file, which can then be transferred to the target host and imported there. Again, as before, the host being registered in the job must already be set up on the site

First, the registration is performed on any host in the Setup by proxy. Here, of course, the Checkmk server comes in handy, as it is usually the first host to be set up. As with the example above, you can provide the password by job option or be asked for it interactively if you omit the --password option. We redirect the JSON output to a file in the example:

Now we transfer the file /tmp/mynewhost3.json to the host we registered for and import the file:

Once the registration is complete, perform a connection test and a service discovery in the Checkmk server Setup. Then, as the last step include the discovered services in the monitoring by activating the changes.

If the connection test fails, refer to the following chapter for testing and troubleshooting information.

You can also deregister a host.

On a host connected to the Checkmk server, you can revoke the trust. Here, in the following command, the Universally Unique Identifier (UUID) to specify is the one output by the cmk-agent-ctl status command:

To delete all connections from the host and additionally restore legacy pull mode, enter the following command:

After that, the agent behaves as it did after the initial installation and before the first registration and sends its data unencrypted.

Complete the deregistration on the Checkmk server: In the Setup, on the Properties of host page, select the Host > Remove TLS registration menu item and confirm the prompt.

In case you prefer the command line: On the Checkmk server, for each connection of a host that is in monitoring, there is a soft link with the UUID that points to the folder with the agent output:

If you delete this soft link, you will need to re-register the host.

To check that the configuration has been read in as expected, call the agent program with the showconfig option. With this option you will not only get the configuration as it is currently used by the agent, but additionally, the environment variables as well as the configuration files in use will always be displayed.

If only a certain part of the configuration is of interest, you can limit the output to that specific part. Here, for example, it is checked whether the options in the ps section have been set correctly:

In this way you get a quick overview of how the three different configuration files have been merged and used by the agent program. Errors will be immediately visible.

The agent program is a simple Windows executable that obtains data from your system and outputs it as loosely-formatted text. You can call this program directly from the command line. With the test parameter it prints everything to standard output. With the help option you get, among other things, a detailed and complete list of options available to you beyond those described here. Since the output can be a bit long, the more pager, with which you can exit from the output with the Q key, is very handy here:

This allows you to test whether the output contains all the desired data — for example, whether all the installed plug-ins provide data.

Since the agent program listens on the local loopback interface (127.0.0.1) on port 50001, you can also test the connection to this using Telnet or Putty.

If registering a host fails even before a certificate is presented, knowledge about the ways of communication can help identifying the problem — and of course solving it.

After entering the cmk-agent-ctl register command, the Agent Controller first asks the Checkmk server for the Agent Receiver port using the REST API. As second step a connection to the Agent Receiver is established to request the certificate. You can simulate the first request on the host with a program like curl:

The parameter --insecure instructs curl to skip the certificate check. This behavior reflects the behavior of the Agent Controller in this step. The response is only a few bytes, containing the port number of the Agent Receiver. For the first site this is usually just 8000, for the second 8001 and so on. Common problems regarding this request are:

When the curl command fails you might change routing or firewall settings to enable access, respectively add the internal CA to the trust chain of the host.

In case the host you are trying to register uses an HTTP proxy, curl will use it, but cmk-agent-ctl won’t do so with default settings. Use the additional Flag --detect-proxy to instruct cmk-agent-ctl to use a proxy configured via system settings.

However often it may be easier to find out the port of the Agent Receiver and note it down. To do so, on the Checkmk server run, logged in as site user:

Now you can specify the port when entering the command for registration. This skips the first request to the REST API. Communication then takes place directly with the Agent Receiver without any detours:

Port 8000 also must be reachable from the host. In case it is not, you will get this error message:

Equivalent to port 443 (respectively 80) mentioned above, you can now adjust routing or firewall settings so that the host to be registered can reach the Checkmk server on the Agent Receiver’s port (8000 or 8001…​)

Should security policies prohibit access to the Agent Receiver, there is still the possibility to use registration by proxy on the Checkmk server.

The Agent Controller provides its own subcommand, dump, which displays the full agent output as it arrives in the monitoring:

This allows you to verify that the data from the agent program has arrived at the Agent Controller. This output does not yet prove that the agent is also accessible over the network.

If you are sure that locally the agent program and the installed plug-ins are executing correctly, you can next check from the Checkmk server via netcat (or nc) whether port 6556 is accessible via the external IP address of the host:

The output 16 indicates whether the connection was successfully established and that the TLS handshake can now take place. Since everything else here is TLS encrypted, no more detailed check is possible.

If the communication between agent and Checkmk server is still unencrypted (as in legacy pull mode), you will get the complete unencrypted agent output with this command instead of the 16.

For more diagnostics to run on the Checkmk server, see the general article on the monitoring agents. In particular, you can also perform a connection test using the Checkmk interface. You will get the result in the Agent: box.

If you get no information or only a timeout error message during the connection test, as in the example above, you should check the Inbound Rules of the Windows firewall on the host.

The agent already creates a rule in the Windows Firewall during its installation, so that the Agent Controller can be reached from the outside via port 6556.

In current versions of Windows you can find the Windows Defender Firewall with Advanced Security via Windows Settings (Settings > Windows Security) or start it by calling wf.msc from the command line:

If you do not find such an entry in the Windows Firewall settings, you can add it at this exact location. To do this, click on New Rule in the Action menu.

This opens a wizard for creating a new firewall rule. Set the five choices as follows:

Alternatively, you can automate this step and set the rule directly on the command line. Modify the following command to your customized installation path if necessary:

Note: The command has been split into four lines for readability.

Security is an important criterion for any software, and of course monitoring is no exception. Since the monitoring agent is installed on every monitored server, a security problem here could have particularly serious consequences.

This is why security was emphasized in the design of Checkmk and has been an absolute principle since the earliest days of Checkmk: The agent does not read data from the network. Period. This means that it is impossible for an attacker to inject any commands or script components via the monitoring port 6556.

For an attacker, however, even a process list can be a first approach to draw conclusions about worthwhile targets. Therefore, transport encryption between agent and Checkmk server with Transport Layer Security (TLS) is mandatory starting with Checkmk version 2.1.0. Here, the Checkmk server "pings" the monitored host, which then establishes the TLS connection to the Checkmk server and transmits the agent output over it. Since only Checkmk servers with which a trust relationship exists can initiate this data transfer, there is no risk of data falling into the wrong hands.

Note: Since there is no Agent Controller, and thus no registry and TLS encryption, on very old Windows systems, you will have to choose other ways of encryption if needed. In this case, we recommend using the built-in (symmetric) encryption with the Encryption (Linux, Windows) rule and/or setting up an SSH tunnel.

Restricting access to specific IP addresses can also be configured via the firewall. However, the agent itself also offers the possibility to simply ignore requests from foreign IP addresses. Just add the following restriction to the configuration file in the global options. Note that there may be other parameters set in the configuration file before or after this and this is just a snippet:

As you can see in the example, you can allow any number of subnets. For example, with /32 you specify a subnet of size 1, so that only this one address is allowed, while with 192.168.42.0/24 you allow all addresses between 192.168.42.0 and 192.168.42.255.

In Agent Bakery, you can configure the allowed IP addresses using the following ruleset: Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Allowed agent access via IP address (Linux, Windows).

Newer versions of Windows (Windows 10 from version 1809 and Windows Server from 2019) have native support for SSH. OpenSSH can be installed here via Apps > Apps & Features > Optional Features. But even with older versions, you can retrofit an SSH server via Cygwin and thus recreate an identical configuration as is possible under Linux. Please note the current help from Cygwin or Microsoft for the setup. Once an SSH server is started and accessible, further setup is identical to that under Linux: You set up authorized_keys on the monitored host and restrict access to running the agent.

However, the entry in the authorized_keys file is a bit bumpy on Windows hosts, as many characters have to be masked here. Use the following (shortened) example for orientation. The forward slashes as path separators are in both examples a permissible and preferred (saves escaping) exception to the otherwise common backslashes:

If the call is to be made under a user with administrator rights, the following two lines in the global configuration file sshd_config must be commented out with hash marks (#) on the Microsoft OpenSSH server:

Note that you can stop the Windows service after this, and as a result any firewall rules that may have been set up will also become obsolete.

When updating the agent, it may happen that the built-in (symmetric) encryption performed by the agent program itself remains active. If TLS encryption and built-in encryption are active at the same time, then the entropy of the transmitted data is so high that compression, which is active from version 2.1.0 onwards, will not save any transmitted data — and will burden the CPUs of both the host and the Checkmk server with additional further encryption and decryption steps.

For this reason, you should disable the built-in encryption promptly after switching to TLS. In the Checkmk Raw Edition you can achieve this by setting the value of the encrypted parameter to no in the configuration file C:\ProgramData\checkmk\agent\check_mk.user.yml.

In the Checkmk Enterprise Editions, you can change existing rules to Use TLS encryption in Setup > Agents > Access to agents > Encryption (Linux, Windows) in the Encryption (Linux, Windows) section and then rebake the agent packages. After the next automatic agent update, the agent program encryption is disabled but guaranteed by the Agent Controller. Note that after the automatic agent update, only registered hosts can provide monitoring data.

The agent program check_mk_agent.exe contains a whole set of sections which provide monitoring data for various check plug-ins which are then automatically found by the service discovery. This includes all important monitoring of the operating system.

In addition, there is the possibility to extend the agent with agent plug-ins. These are small scripts or programs that are called by the agent and extend it with additional sections with additional monitoring data. The Checkmk project provides a number of such plug-ins, which — if installed and configured correctly — automatically provide new services in the service discovery.

Why aren’t these plug-ins simply hard-coded into the agent? For each of the plug-ins there is one of the following reasons:

The included plug-ins for Windows can all be found on the monitored host in the agent installation directory under C:\Program Files (x86)\checkmk\service\plugins. They are stored there so that they are directly available.

Alternatively, the plug-ins for Windows are also located on the Checkmk server under share/check_mk/agents/windows/plugins. They are also available from the agent download page in the Setup menu (as described in the Installation chapter) in the Plugins box:

For all agent plug-ins we provide, there are matching check plug-ins that can evaluate their data and generate services. These are already installed, so that newly found services are immediately recognized and can be configured.

Note: Before you install a plug-in on the host, take a look at the corresponding file. Often you will find important information there about the correct use of the plug-in.

The actual installation is then simple: Copy the file to C:\ProgramData\checkmk\agent\plugins.

Once the plug-in is in the correct directory, it will be called automatically by the agent and a new section will be created in the agent output. This usually has the same name as the plug-in. Complex plug-ins (e.g. mk_oracle.ps1) even create a whole set of new sections.

Some plug-ins need a configuration file in C:\ProgramData\checkmk\agent\config to work. For others, a configuration is optional (e.g. mssql.vbs) and allows special features or customizations. Still others just work that way. You have several sources to get information:

For special (scripting) languages, it may be necessary to enable them first in the agent configuration. For example, Python scripts will not be executed unless they are explicitly enabled. You can do this by extending the file extensions in the check_mk.user.yml configuration file in the global section, as shown in the following excerpt:

Important: The use of such plug-ins requires that the files can also be called in a regular command line without special paths. In the case of Python, it must be installed correctly and the path to the interpreter must be present in the environment variables. Instructions on how to set up Python correctly can be found directly on the pages of the Python Software Foundation.

Each plug-in can be executed in different modes. The following options are available for entry in the configuration file. The value in bold is the default value:

A configuration for the Veeam plug-in looks then for example like this (the excerpt is shortened and contains only the relevant part for the example):

According to the above exemplary configuration the plug-in in the data directory C:\ProgramData\checkmk\agent\plugins is executed asynchronously every five minutes (300 seconds) and may run for a maximum of two minutes (120 seconds). If the plug-in runs into this timeout, it will try a second time to get a result.

In the Checkmk Enterprise Editions, the included plug-ins can be configured via the Agent Bakery. This takes care of both the installation of the plug-in itself and the correct creation of the configuration file, should one be needed.

Each plug-in is configured via an agent rule. You can find the appropriate rule sets in Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Agent Plugins:

Since agent plug-ins are executable programs, you can run them manually for testing and diagnostic purposes. However, there are plug-ins that need certain environment variables set by the agent to find their configuration file, for example. If necessary, set them manually if they are needed in the script or program.

There are two good reasons to use Nagios plug-ins also under Checkmk. If you migrated your monitoring from a Nagios based solution to Checkmk, you can continue to use older check plug-ins for which there is no Checkmk counterpart yet. In many cases these are self-written plug-ins in Perl or Shell.

The second reason for using Nagios plug-ins is true end-to-end monitoring. Let’s assume you have your Checkmk server, a web server and a database server distributed over a large data center. In such a case, the response times of the database server measured from the Checkmk server are not very meaningful. It is far more important to know these values for the connection between the web server and the database server.

The Checkmk agent provides a simple mechanism to meet both of these requirements: MK’s Remote Plugin Executor or MRPE for short. The name is deliberately an analogy to Nagios' NRPE, which performs the same task there.

The MRPE is built into the agent and is controlled by various configuration files.

As an alternative to configuring directly on a host in the user-specific configuration file, you can also define your MRPE plug-ins directly in the Setup menu. To do this, use the Setup > Agents > Windows, Linux, Solaris, AIX > Agent > Agent rules > Execute MRPE checks rule set. The necessary entry is then automatically created in the Agent Bakery configuration file.