Alert handlers

One would think it obvious that a monitoring system should never intervene in events — but rather that it should, well, monitor. And it’s probably a good idea to leave it at that.

It is however an admittedly attractive notion that a system which can reliably identify problems could also correct these, provided that it could function automatically.

A few suitable examples can be easily imagined:

If one can accept this, then one must think differently about monitoring. From a system that simply observes and is 'not necessary' for operations, a step by step process leads to monitoring becoming a vital organ in the data center.

But correcting problems is not the only thing that monitoring can do automatically when it identifies a problem. Very useful, but also harmless is the collection of additional diagnostic data at the moment of a failure. You could no doubt offhand think of numerous further issues with which one could use alert handlers as a starting point.

Alert handlers are scripts you write yourself, that are run for you by Checkmk in the commercial editions if a problem is detected - or more precisely — if a host or service changes its status.

Alert handlers are very similar to notifications and are similarly configured, but there are a few important differences:

One can also say that alert handlers are very 'low level'. As soon as a host or service changes its status, your configured alert handlers will be immediately activated. In this manner an alert handler can even perform a repair successfully before an actual alert is generated.

You can naturally - as always in Checkmk - use rules to define the conditions for which a particular handler should be executed. You can find out how to do this and everything else about alert handlers in this article.

A tip for Checkmk Raw users: you can also have the monitoring execute actions automatically. Use Nagios’s 'event handlers' for this. Configure this with manual configuration files in Nagios syntax under ~/etc/nagios/conf.d/. The event handlers are well documented. Information can be found simply via Google.

Alert handlers are scripts that are run on the Checkmk server. They must be stored in the ~/local/share/check_mk/alert_handlers/ directory, and may be coded in any Linux-supported language, e.g., BASH, Python or Perl. Do not forget to make the scripts executable with chmod +x.

If a comment is inserted into the second line of the script (with a # hash), this will appear as the script’s name in the rule’s selection list:

As with notifications the script obtains all information for the host or service as environment variables, all of which begin with the ALERT_ prefix.

To test exactly which environment variables appear in the script, you can use the following alert handler for a test:

The handler activation is performed using Setup > Events > Alert handlers.

Proceed as follows:

The form for selecting the alert handler allows direct access and shows the title which is logged in the second line of the script. Additionally you may add arguments, which you enter in the text fields. These will be interpreted as command line arguments in the script. On your shell you may access them with $1, $2, etc.

Important: After saving the rule the alert handler will be immediately active and will run with every change of status for any host or service!

To test, manually set a service, for example, Fake check results to CRIT. Now the file should have been created with the variables. Here are its first twenty lines:

A log file for the alert handler’s (non-)execution will be found in ~/var/log/alerts.log. The section for the execution of handler debug, for the service Filesystem / on host myserver123 will look something like this:

A couple of further useful tips:

Alert handlers are very often used to log in to an affected machine remotely using SSH or another protocol, and once there execute a script-controlled action. Since this machine is experiencing a problem, it cannot be excluded that connecting takes a long time or even goes into a timeout.

So that the monitoring is not allowed to stand still, or other alert handlers stalled during this time, as a matter of principle alert handlers are run asynchronously. An auxiliary process - the alert helper — is responsible for this function, and it is started by the CMC. To reduce overhead, this only occurs if a minimum of one alert handler rule has been created. In the cmc.log you will then see the following line:

With every change of state for a host or service, the alert helper receives a notification from the CMC containing all information relevant to the event. It then evaluates all alert rules and determines whether a handler should be triggered. If yes, the appropriate script will be started and run in the background as an external process.

When you stop the CMC (e.g. via omd stop or by shutting the monitoring server down), all alert helpers still running will be aborted. These will not be repeated later - since who knows when 'later' will be? It’s possible that restarting a service or similar may be more damaging than useful!

In order to protect itself from too many processes being started in the case of an error situation, when an alert handler is running a 60 second (settable) timeout is in force. At the end of this time the handler will be stopped. In detail, this means at the end of a timeout a Signal 15 (SIGTERM) will be sent to the handler. In this way it has the possibility to stop itself cleanly. After a further 60 seconds (double timeout) it will then be 'terminated' finally with a Signal 9 (SIGKILL).

Checkmk precludes the simultaneous running of alert helpers if they apply to the same host/service and would run the same script with the same parameters. Such a situation indicates that the first handler is still running and that it would make no sense to start a second copy of the same handler — the second handler would be instantly canceled and identified as 'failed'.

The alert handler’s outputs and exit codes are reliably evaluated and returned to the core where it is saved in the monitoring history. Furthermore you can trigger a notification (see below).

There are a number of global settings for executing alert handlers:

The Alert handler log level influences the logging in the alert helper log file (~/var/log/alerts.log).

With a click in the Master control snap-in you can deactivate alert handlers globally. Currently running handlers will not be affected and will run to completion.

Do not forget to reset the small switch to green as soon as practicable! Otherwise you could be deceived by a false sense of security that the monitoring is fixing everything…​

Every Checkmk version includes a built-in alert handler that enables the reliable execution of scripts on monitored Linux systems. The most important features of this solution are:

The Linux remote alert handlers consists of the following individual elements:

Assuming one wants to run the /etc/init.d/foo restart script on the myserver123 Linux system whenever the Process FOO service becomes critical (which we have already set up). Proceed as follows:

Of course manually preparing an agent also works. In such a case we recommend performing the Agent Bakery procedure on a test system, then examine the relevant data and replicate it manually on your own system. A list of the file paths can be found here.

Here in this case it is also important that in the Agent Bakery you create a rule for installing the remote handler — because in this rule the SSH keys will be generated for the access and also for use by the alert handler! The public key for the installation in authorized_keys can be found in the ~/etc/check_mk/conf.d/wato/rules.mk configuration file (or in a subfolder in rules.mk).