The Event Console

As always in Checkmk, you must first activate changes in order for them to take effect. This is not a disadvantage because this way, for changes that affect several related rules, you can specify exactly when the rules should go 'live'. And beforehand you can use the Event Simulator to test if everything fits as intended.

First, click on the number of accumulated changes at the top right of the page.

Then click Activate on selected sites to activate the change. The Event Console is designed in such a way that this action runs with no interruptions. The reception of incoming messages is ensured at all times, so that no messages can be lost in the process.

Only administrators are allowed to activate changes in the EC. This is controlled via the Activate changes for event console permission.

For testing you could of course now send messages via Syslog or SNMP. You should also do this later. But for a first test the EC’s built-in Event Simulator is more practical:

Here you have two options: Try out evaluates on the basis of the simulated message which of the rules would match. If you are in the top level of the Setup GUI for the EC, the rule packages will be highlighted. If you are inside a rule package, the individual rules are highlighted. Each package or rule is marked with one of the following three symbols:

Clicking Generate event does almost the same as Try out, only now the message is actually generated. Any defined actions are actually executed. And the event will then also show up in the open events in the monitoring. You can see the source code for the generated message in the confirmation:

The generated event appears in the Monitor menu under Event Console > Events:

For a first real test over the network, you can easily send a syslog message from a another Linux computer by hand. Since the protocol is so simple, you don’t even need a special program, you can simply send the data via netcat or nc using UDP. The content of the UDP packet consists of a single line of text. If this conforms to a specific structure, the Event Console breaks down the components cleanly:

But you can also simply send anything. The EC will then accept it anyway and just evaluate it as a message text. Additional information such as the application, the priority, etc. will of course be missing. For security reasons the status CRIT will be assumed:

Within the Checkmk site running the EC there is a named pipe, into which you can write text messages locally via echo. This is a very simple method for connecting a local application and also a way to test the processing of messages:

By the way, it is also possible to send in syslog format here, so that all fields in the event data are filled in cleanly.

The Update & Acknowledge command displays the following area above the event list:

With the Update button you can in a single action add a comment to the event, add a contact person and acknowledge the event. The Change contact field is intentionally free text. Here you can also enter things like phone numbers. In particular, this field has no influence on the visibility of the event in the GUI — it is purely a comment field.

The Set event to acknowledged checkbox will cause the event to switch from the open phase to acknowledged, and henceforth be displayed as handled. This is analogous to the acknowledgment of host and service problems.

Subsequent invocation of the command with the checkbox unchecked removes the acknowledgment.

The Change State command allows the manual changing of an event’s state — e.g. from CRIT to WARN.

With the Custom Action command, freely definable actions can be executed on events. Initially, only the Send monitoring notification action is available. This will send a Checkmk notification which will be handled in the same way as one from an actively monitored service. This goes through the notification rules and can generate emails, an SMS, or whatever you have configured accordingly. See below for details on notification by the EC.

The Archive Event button permanently removes an event from the open events list. Since all actions on events — including this deletion — are also recorded in the archive, you will still be able to later access all of the information on the event. That is why we don’t speak of deleting, but rather of archiving.

You can also conveniently archive individual events from the event list using .

To provide visibility of hosts and services in monitoring for normal users, Checkmk uses contact groups. These groups are assigned to the hosts and services through Setup GUI, rule or folder configuration.

With the Event Console the situation is that such an assignment of events to contact groups does not exist. This is because it is simply not known in advance which messages will actually be received. Not even the list of hosts is known, because the sockets for Syslog and SNMP are accessible from everywhere. Therefore the Event Console includes a few special features for defining visibility.

First of all, when configuring the user roles there is the Event Console > See all events permission. This is active by default, so that normal users are allowed to see all events! This is deliberately set to ensure that important error messages do not fall by the wayside due to incorrect configuration. The first step to more precise visibility is to remove this permission from the user role.

To ensure that the visibility of events is as consistent as possible with the rest of the monitoring, the Event Console tries as much as it can to match the hosts from which it receives events to those hosts configured via the Setup GUI. What sounds simple is tricky in detail. Sometimes a host name is missing in the event and only the IP address is known. In other cases the host name is spelled differently to what is in the Setup GUI.

The assignment is done as follows:

You can influence the assignment at one point: Namely, if contact groups are defined in the rule and the host could be assigned, the mapping usually has priority. You can change this in a rule with the Precedence of contact groups option:

In addition, you can make settings directly in the rule for the notification. This makes it possible to prioritize the type of event over the regular responsibilities for a host.

Both for the rule packs  …​

…  as well as for the individual rules  …​

… in the Hits column you will find the information on how often the package or rule has already matched a message. This helps you to eliminate or repair ineffective rules. But this can also be interesting for rules that match very often. For optimal performance of the EC, these should be at the beginning of the rule chain. This way you can reduce the number of rules that the EC has to test on every message.

You can reset the counters at any time with the Event Console > Reset counters menu item.

At Trying out a rule you have already seen how you can use the Event Simulator to check the evaluations of your rules. You can get similar information at runtime for all messages, if in the Event Console settings you set the value from Debug rule execution to on.

The Event Console log file can be found at var/log/mkeventd.log, in which you will find the exact reason why any rule that was checked did not take effect:

It goes without saying that you should use this intensive logging only when needed and with caution. In an only slightly more complex environment huge volumes of data will be generated!

For all conditions involving text fields, the comparison text is always treated as a regular expression. The comparison always takes place ignoring upper/lower case. The latter is an exception to Checkmk conventions in other modules. However, this makes the formulation of the rules more robust, especially since host names in events are not necessarily consistent in their spelling if they have been configured on each host locally rather than centrally. This exception is thus very useful here.

Furthermore, an infix match always applies — i.e. a check for a content of the search text. So you can save a .* at the beginning or at the end of the search text.

There is one exception however: If no regular expression is used in the match to the host name, rather an explicit host name, it will be checked for an exact match and not for being contained.

Attention: If the search text contains a dot (.), it will be considered a regular expression and the infix search will apply, for example myhost.com will then also match e.g. notmyhostide!

Very important and useful here is the concept of match groups in the Text to match field. This refers to the text sections that match parenthesized expressions in the regular expression.

Suppose you want to monitor the following type of message in the log file for a database:

The WP41 is of course variable and you surely do not want to formulate a separate rule for each different instance. Therefore you use .* in the regular expression, which stands for any string:

Database instance .* has failed

If you now enclose the variable part in round brackets, the Event Console will memorize (capture) the actual value for any future actions:

Database instance (.*) has failed

After a successful match of the rule, the first match group is now set to the value WP41 (or whichever instance produced the error).

You can see these match groups in the Event Simulator if you move the cursor over the green ball:

You can also see the groups in the details of the generated event:

Among other things, the match groups are used for:

At this point, a tip: There are situations where you need to group something in the regular expression but you do not want to create a match group. You can do this by placing a ?: directly following the opening parenthesis. Example: The expression one (.*) two (?:.*) three creates for a match on one 123 two 456 three only the one match group 123.

In the Match original source IP address field you can match the IPv4 address of the message sender. Specify either an exact address or a network in the notation X.X.X.X/Y, e.g. 192.168.8.0/24 to match all addresses in the network 192.168.8.X.

Note that the match on the IP address only works if the monitored systems send directly to the Event Console. If another connected intermediate syslog server is forwarding the messages, its address will appear instead as the sender in the message.

The Match syslog priority and Match syslog facility fields are standardized information, originally defined by Syslog information. Internally, an 8-bit field is divided into 5 bits for the facility (32 possibilities) and 3 bits for the priority (8 possibilities).

The 32 predefined facilities were once meant for something such as an application. But the selection was not made very forward-looking at that time. One of the facilities is uucp — a protocol which by the early 90s of the last millennium was already nearly obsolete.

But it is a fact that every message that comes via syslog carries one of these facilities. In some cases you can also assign these freely when sending the message, in order to be able to filter them later. This is quite useful.

The use of facility and priority also has a performance aspect. If you define a rule that in any case only applies to messages that all have the same facility or priority, you should define these additionally in the rule’s filters. The Event Console can then bypass these rules very efficiently when a message with different values is received. The more rules that have these filters set, the fewer rule comparisons are needed.

The Negate match: Execute this rule if the upper conditions are not fulfilled. checkbox causes the rule to take effect only when none of the conditions are fulfilled. This is actually only useful in the context of two rule types (Rule type in the Outcome & Action box of the rule):

You can learn more about the rule packs below.

When a rule matches, it specifies what should happen with the message. This is done in the Outcome & Action box:

The Rule type can be used to abort the evaluation at the point entirely or for the current rule package. Especially the first option should be used to get rid of most of the useless 'noise' using a few specific rules at the very beginning. Only with the 'normal' rules are the other options in this box actually evaluated.

With State the rule sets the status of the event in the monitoring. In the rule this state will be WARN or CRIT. Rules that generate OK events can be interesting in exceptions to represent certain events in a purely informative way. In such a case a combination with an automatic expiration of these events is then interesting.

As well as setting an explicit status, there are two more dynamic options. The (set by syslog) setting takes over the classification based on the syslog priority. This only works if the message has already been classified usable by the sender. Messages received directly by syslog will contain one of eight RFC priorities, which are mapped as follows:

In addition to syslog messages, messages from the Windows Event Log, and messages from text files that have already been classified with the Checkmk logwatch plug-in on the target system, provide ready-made states. For SNMP traps this is unfortunately not available.

A completely different method is to classify the message based on the text itself. This can be done with the (set by message text) setting:

The match to the texts configured here happens only after checking for Text to match and the other rule conditions have been checked, so that you do not have to repeat those checks.

If none of the configured patterns is found, the event returns the UNKNOWN state.

The idea behind the Service Level field is that every host and every service in an organization has a certain level of importance. This can be associated with a specific service agreement for the host or service. In Checkmk, you can use rules to assign such service levels to your hosts and services and then, for example, make the notification or self-defined dashboards depend on it.

Since events do not necessarily correlate with hosts or services, the Event Console allows you to assign a service level to an event using a rule. You can then later filter the event views using this level.

By default, Checkmk defines four levels: 0 (no level), 10 (silver), 20 (gold), and 30 (platinum). You can change this selection as required in the Global settings > Notifications > Service Levels. Decisive are the numbers designating the levels, since the levels are sorted by these numbers and also compared according to the relative importance.

The contact groups used for the visibility will also be used for the notification of events. Here you can use rules to assign contact groups explicitly to events. Details can be found in the chapter on monitoring.

Actions are very similar to Alert Handlers for hosts and services. Here you can have a self-defined script executed when an event is opened. Full details describing the actions can be found below in a separate section.

Automatic deletion (= archiving), which you can set with Delete event immediately after the actions ensures that an event is not visible in the monitoring at all. This is useful if you only want to trigger some actions automatically or if you only want to archive certain events so that you can search for them later.

You can also have actions executed automatically when an event is canceled. It is important to know that when an event is canceled, a number of data fields in the event are overwritten by values from the OK message before any actions are executed. In this way the full data of the OK message is then available in the action script. Additionally, during this phase the event’s state is flagged as OK. In this way an action script can detect an override and you can use the same script for error and OK messages (e.g. when connecting to a ticket system).

The following fields are overwritten from OK message data:

All other fields remain unchanged — including the event ID.

If you are using rewrite and cancel in the same rule, you should be careful when rewriting the host name or the application. When canceling, the EC always checks whether the canceling message matches the host name and application of the open event. But if these were rewritten the cancellation would never work.

Therefore, before canceling the event, the Event Console simulates a rewrite of the host name and application to compare the relevant texts. This is probably what you would expect.

You can also take advantage of this behavior if the application field in the error message and the later OK message do not match. In this case, simply rewrite the application field to a known fixed value. This in fact leads to the result that this field will be ignored.

There are (unfortunately) situations where the text of the error and OK messages is absolutely identical. In most cases the actual status is not coded in the text, but in the in the syslog priority.

For this purpose there is the Syslog priority to cancel event option. Here enter the range debug …​ notice for example. All priorities in this range are normally considered to have an OK status. When using this option you should still enter a suitable text in the Text to cancel event(s) field, otherwise the rule will match on all OK messages that affect the same application.

You can activate the check for messages that occur too often with the Count messages in interval option:

Here you first specify a time period at Time period for counting and the number of messages which should lead to the opening of an event at Count until triggered. In the example above this is set to 10 messages per hour. Of course these are not 10 arbitrary messages, rather messages that are matched by the rule.

Normally it makes sense not to count all matching messages globally, but only those which refer to the same 'cause'. To control this, there are the three checkboxes for options prefixed by Force separate events for different …​. These are preset in such a way that messages are only counted together if they match in:

This allows you to formulate rules such as 'If more than 10 messages are received per hour from the same host, the same application and there the same site, then…​'. Due to the rule this can result in multiple differing events being generated.

If, for example, you deselect all three checkboxes, then the counting will be done only globally and the rule will be able to generate only one event in total!

By the way, it can make sense to enter 1 as the count. In this way you can effectively get a grip on 'event storms'. If, for example in a short time frame 100 messages of the same kind are received, then nevertheless only one event will be created. You will then see in the event details

When the case is then closed, via two checkboxes you can define when a new event should be opened. Normally an acknowledgment of the event creates a situation that if further messages are received, a new count is started and a new event is triggered. You can disable this function with Continue counting when event is acknowledged.

The Discontinue counting after time has elapsed option ensures that a separate event is always opened for each comparison period. In the above example a threshold of 10 messages per hour was specified. If this option is enabled, a maximum of 10 messages of one hour will be added to an already opened event. As soon as the hour has elapsed, if a sufficient number of messages have been received a new event will be opened.

For example, if you set the number to 1 and the time interval to one day, you will only see a maximum of one event of this message type per day.

The Algorithm setting may be a bit surprising at first sight. But let’s be realistic: What do you actually mean by '10 messages per hour'? Which hour is meant by that? Always full hours of the time of day? It could be that in the last minute an one hour nine messages are received, and nine more in the first minute of the next hour. That then makes 18 messages in only two minutes of elapsed time, but still less than 10 per hour, and so the rule would not apply. That doesn’t sound very reasonable  …​

Since there is no single solution to this problem, Checkmk provides three different definitions of exactly what '10 messages per hour' should mean:

So which algorithm should you choose?

Events that have not yet reached the set number, are latent but not automatically visible to the operator. They are in the counting phase. You can make such events visible with the phase filter in the event view:

Just as the arrival of a particular message can mean a problem, so too the absence of a message can also mean a problem. You may expect at least one message per day from a particular job. If this message does not arrive, the job is probably not running and should be fixed urgently.

You can configure something like this under Counting & Timing > Expect regular messages:

As with counting, you need to specify a time period in which you expect the message(s) to appear. Here, however, a completely different algorithm is used, which makes much more sense at this point. The time period is always aligned exactly to defined positions. For example, the interval hour always starts at minute and second zero. You have the following options:

Why is this so complicated? This is to avoid false alarms. Do you expect one message from a backup per day for instance? Surely there will be slight differences in the runtime of the backup, so that the messages are not received exactly 24 hours apart. For example, if you expect the message to arrive around midnight, give or take an hour or two, an interval from 12:00 to 12:00 will be much more realistic than one from 00:00 to 00:00. However, should the message fail to appear you will only receive a notification at 12:00 o’clock midday.

The Merge with open event option is preset in such a way that in the case of the same message appearing repeatedly, the existing event will be updated. You can alter this so that a new event is opened each time.

With the Add new action button you can create a new action. The following example shows how to create a simple shell script as an action of the Execute Shell Script type. Details for the events are available to the script via environment variables, for example the $CMK_ID of the event, the $CMK_HOST, full text $CMK_TEXT or the first match group as $CMK_MATCH_GROUP_1. For a complete list of available environment variables, see the inline help.

Older versions of Checkmk allowed environment variables as well as macros such as $TEXT$ which were replaced before the script was executed. Because of the danger that an attacker could inject commands via a specially crafted UDP packet that is executed with the privileges of the Checkmk process, you should not make use of macros. Macros are currently still supported for for compatibility reasons, but we reserve the right to remove them in a future version of Checkmk.

The sample script shown in the screenshot creates the file tmp/test.out in the site directory in which it writes a text containing the concrete values for the variables for the respective last event:

The scripts are executed under the following environment:

If your script possibly contains wait times, you can make it run asynchronously using the at Linux spooler. To do this, create the script in a separate local/bin/myaction file and start it with the at command, e.g.:

The Send Email action type sends a simple text email. You could actually also do this with a script, for instance by using the mail command line command. But this way it is more convenient. Note that placeholders are also allowed in the Recipient email address and Subject fields.

Events can come from any host — regardless of whether they are configured in the active monitoring or not. After all, the syslog and SNMP ports are open to all hosts on the network. As a result the extended host attributes such as alias, host tags, contacts, etc. are initially not available. This in particular is why conditions in notification rules do not necessarily work as you would expect them to.

For example, when notifying, the EC attempts to find a host within the active monitoring that matches the event. This is done using the same procedure is used as that for visibility of events. If such a host can be found, the following data is taken from this host:

As a result, the host name in the processed notification may not exactly match the host name in the original message. The coding of notification texts that conform with those from the active monitoring, however, simplifies the formulation of uniform notification rules which include conditions that apply to the host name.

The mapping is done in real time by sending an Livestatus query to the monitoring core which runs in the same site as the EC that received the message. Of course this only works if the syslog messages, SNMP traps etc. are always sent to the Checkmk site on which the host is being actively monitored!

If the query does not work or the host cannot be found, substitute data will be accepted:

If the host cannot be assigned in the active monitoring, this can of course lead to problems with the notifications. On the one hand because of the conditions, which may then no longer apply, and also because of the contact selection. For such cases you can modify your notification rules so that notifications from the Event Console are handled specifically using their own rules. For this purpose there is a separate condition with which you can either match positively only to EC notifications, or vice versa exclude them:

In order for notifications from the EC to pass through the active monitoring’s notification system, the EC must be adapted to to conform to its scheme. In the process the typical data fields for a notification are filled as effectively as possible. We have just described how the host’s data is determined. Further fields are:

As described above, it may not be possible to determine the contacts for an event automatically. In such cases you can specify contact groups directly in the EC rule which is to be used for the notification. It is important that you do not forget to check the Use in notifications box:

There is a central switch for notifications in the Master control snap-in. This also applies to notifications forwarded by the EC:

Just as with the host assignment, querying the switch by the EC requires Livestatus access to the local monitoring core. A successful query can be seen in the log file from the Event Console:

The Event Console is able to detect hosts that are currently in a scheduled downtime and does not send notifications in such situations. In the log file it will look like this:

Of course, this also requires the host to be successfully found in the active monitoring. If this is not successful, it is assumed that the host is not in downtime and a notification will in any case be generated.

When you write your own notification scripts, especially for notifications coming from the Event Console a number of additional variables will be available that describe the original event (accessed as usual with the prefix NOTIFY_):

To measure your performance there is a separate snap-in for the sidebar with the name Event Console Performance. You can include this as usual with :

The values shown here are averages over about the last minute. An event storm that lasts only a few seconds cannot be read here directly, but the numbers have been somewhat smoothed out and are thus easier to read.

To test the maximum performance, you can artificially generate a storm of unclassified messages (please only in the test system!), for example, in a shell loop by continuously writing the contents of a text file into the events pipe.

The most important readings from the snap-in have the following meaning:

You can see how many messages the Event Console can process per second from the Processing time per message. This time is generally related to the number of rules that have to be tried before a message is processed. You have several options for optimization here:

Furthermore, there is an optimization in the EC based on the syslog priority and facility. For each combination of priority and facility, a separate rule chain is created internally in which only such rules that are relevant for messages of this combination are included.

Each rule that contains a condition on the priority, the facility or preferably both, will not be included in all of these rule chains, instead optimally only in one. This means that the rule does not need to be checked for messages with a different syslog classification.

Following a restart, in ~/var/log/mkeventd.log you will see an overview of the optimized rule chains:

In the above example you can see that in any case there are 67 rules that must be checked. For messages from the daemon facility 89 rules are relevant, only for the combination daemon/emerg 120 rules have to be checked. Each rule which gets a condition on priority or facility further reduces the number from 67.

Of course you can set these conditions only if you are sure that they are fulfilled by the relevant messages!

The Event Console has an automatic protection against an 'overflow'. This limits the number of current events per host, per rule and globally. Not only open events are counted, but also events in other phases, such as delayed or counting. Archived events are not counted.

This will protect you in situations where, due to a systematic problem on your network, thousands of critical events are flooding in and the Event Console would 'shut down'. On the one hand, this prevents a performance degradation of the Event Console, which would have to hold too many events in its main memory. On the other hand the overview for the operator is (somewhat) protected and events that are not part of the storm remain visible.

Once a limit is reached, one of the following actions will occur:

The limits as well as the associated effect on reaching them are defined in the Event Console settings with Limit amount of current events. The following figure shows the default setting:

If you have enabled a value with …​create overflow event, when the limit is reached, a single artificial event will be created, informing the operator of the error situation:

If you have additionally enabled a value with …​notify contacts, the appropriate contacts will be notified via Checkmk notification. That notification will go through the Checkmk notification rules. These rules do not have to follow the Event Console’s contact selection, but can modify it. The following table shows which contacts are selected if you have set Notify all contacts of the notified host or service (which is the default):