 |
This article is incomplete as it is currently being revised for publication.
Please be patient. Thank you! |
1. Introduction
2. The central notifications view
2.1. Global and predefined notification rule
The last part of the overview page consists of the existing global rule.
If you have a freshly-installed Checkmk, precisely one rule will have been predefined:
Such a rule is structured as follows:
Condition |
All status changes of hosts to DOWN and UP, and of services to CRIT, WARN and OK. |
Method |
Sends an email in HTML format (with embedded metric graphs). |
Contact |
All contacts of the affected host/service. |
As usual, you can edit the rule 
, clone it 
, delete it 
, or create a new rule.
Once you have more than a single rule, you can change their processing order by dragging and dropping them with the 
icon.
|
Changes to notification rules do not require activation of the changes, rather they take effect immediately. |
3. Simple notifications by email
An email notification sent by Checkmk in HTML format looks something like this:
As can be seen in the example, the email also contains the current metrics for the affected service.
Before you receive such an email from Checkmk, a few preparations are necessary, as are described below.
3.1. Prerequisites
In the default Checkmk configuration, a user will receive notifications per email when the following prerequisites have been satisfied:
The Checkmk server has a functioning setup for sending emails.
An email address is configured for the user.
The user is a member of a contact group, and is therefore a contact.
A monitoring event occurs on a host or service assigned to this contact group, which triggers a notification.
3.2. Setting up mail dispatching in Linux
For the successful sending of emails, your Checkmk server must have a functioning SMTP-server configuration. Depending on your Linux distribution, this could utilize, for example, Postfix, Qmail, Exim, or Nullmailer. The configuration will be implemented with your Linux distribution’s resources.
The configuration is generally confined to registering a ‘smarthost’ (also known as an SMTP relay server) to which all emails will be directed. This will then be your firm’s internal SMTP mail server. As a rule smarthosts don’t require authentication in a LAN — which makes matters simple. In some distributions the smarthost will be queried during the installation. With the Checkmk appliance one can configure the smarthost conveniently via the web interface.
You can test the sending of emails easily with the mail command on the command line.
Because there are numerous different implementations for this command under Linux, for standardization Checkmk provides the version from the Heirloom mailx project directly in the site user’s search path (as ~/bin/mail).
The corresponding configuration file is ~/etc/mail.rc.
The best way to test this is as a site user, since the notification scripts will later run with the same permissions.
The email’s content is read from the standard input, the subject specified with -s, and the recipient’s address simply appended as an argument to the end of the command line:
OMD[mysite]:~$ echo "content" | mail -s test-subject harry.hirsch@example.comThe email should be delivered without delay.
If this doesn’t work, information can be found in the SMTP server’s log file in the /var/log directory (see Files and directories).
3.3. Email address and contact groups
A user’s email address and contact groups are defined in the user administration:
In a freshly-generated Checkmk site, initially there is only the Everything contact group. Members of this group are automatically responsible for all hosts and services, and will be notified of every relevant monitoring event by email.
3.4. Special cases: Ticket system, messenger and event engine
Instead of email or SMS, you can also send notifications to a ticket system (such as Jira or ServiceNow), a messenger (Slack, Mattermost) or an event engine (Event Console).
There is a separate notification method for each of these special cases, which can be selected in the notification rule. However, you must note the following two points when creating the rule:
When selecting contacts, make sure that the notifications are only sent to one contact, e.g. by selecting a single user. With the notification methods for ticket systems etc., the contact selection only serves to specify that notifications are sent. However, the notifications are not sent to the selected user, but to the ticket system. Note that a contact selection via contact groups, all contacts of an object or similar usually generates several identical notifications for an event, which then end up in the ticket system twice, three times or even more often.
If the first point is fulfilled, but the user is used in several notification rules for the same method, then only the last rule applies in each case. It is therefore advisable to create a separate functional user for each of these notification rules.
3.5. Fine-tuning HTML email
When sending HTML emails, you may wish to add additional information or flexibly define a reply address (Reply to) to a specific contact for any queries. For this, there is the Setup > Notifications > Parameters for notification methods > HTML Email rule, and in notification rules the HTML email notification method. With these rules you can add a number of parameters such as response address, additional fields with details or free text formatted as HTML.
Note that in the Custom HTML section (e.g. title, description…) field, for security reasons only a small set of HTML tags is allowed. These are:
| Tag | Function | Tips |
|---|---|---|
|
Permitted when combined with the |
|
|
||
|
||
|
||
|
Deprecated. Do not use this tag! |
|
|
||
|
||
|
||
|
Deprecated. Do not use this tag! |
|
|
Spaces and indentations are preserved. |
|
|
||
|
||
|
||
|
Use only within the following lists |
|
|
||
|
As usual with all rules in Checkmk, a very finely granulated application is possible, so that you can individualize a detailed set of notifications to hosts and services as required.
4. Controlling notifications with rules
4.1. Structure of the notification rules
In the following, we present the general structure of the notification rules with the definitions of conditions, event types, filters, notification methods, contacts and general properties.
Conditions
Conditions determine when a rule will be used. They are the basis for every rule. For comprehension it is important to remember that the source is always a monitoring event on a concrete host or service.
The conditions address
the object’s static attributes – e.g., whether the service name contains the
/tmptext or whether a host is in a specific host group,with the current state or the change of the state, e.g. whether the service has just changed from OK to CRIT,
or with completely different things, e.g., whether the 'working time' time period is currently active.
There are two important points to consider when setting the conditions:
If no conditions have been defined the rule will take effect for every monitoring event.
As soon as you select even a single condition, the rule takes effect only if all conditions are met. All selected conditions are linked with AND. There is only one exception to this important rule, which we will discuss later and not consider now.
This means that you should pay very close attention to whether the conditions you have chosen can be met at the same time so that a notification is triggered for the desired case.
The exception to the AND operation (part 1)
The following generally applies in Checkmk: Only if a monitoring event satisfies all configured conditions, the notification rule will apply. There is one important exception to this general rule: for the Host events and Service events conditions.
Should you select only Host events, the rule will match no single service event. Similarly, this applies to the selection of Service events and host events. Should you activate both conditions however, the rule will match if an event is defined in any of the two two conditions. In this exceptional case these conditions will thus not be linked with a logical AND, but rather with an OR. In this way you can simply administer host and service notifications with a single rule.
The exception to the AND operation (part 2)
A further tip regarding the Contact group filters option in the image above: The condition checked here is whether the host/service in question has a certain contact assignment. This can be used to implement functions such as "Host notifications in the contact group Linux should never be sent by SMS". This has nothing to do with the contact selection described above.
Notification method
The notification method in step 3 specifies the technique to be used for sending the notification., e.g. with HTML email.
Each method is realized using a script. Checkmk includes a number of scripts. You can also quite easily write your custom scripts in any desired programming language in order to implement special notifications — for example, to redirect a notification to your own ticket system.
A method can include parameters — such as allowing the method that sends ASCII and HTML emails to explicitly set the sender’s address (From:) for example.
Before making settings directly in the notification rule, one should know that parameters for the notification methods can also be specified via rules for hosts and services: Under Setup > Services > Service monitoring rules, in the Notifications section, you will find a rule set for each notification method, which you can use to define the same settings — and as usual, can be dependent on the host or service.
Parameter definitions in notification rules enable these settings to be varied in individual cases. So you can, for example, define a global subject for your email, but also define an alternative subject with an individual notification rule.
Instead of parameters you can also select Cancel previous notifications – with which all of this method’s notifications from prior rules will be deleted. For more on this, see the topic deleting notifications.
|
For many notification methods for forwarding to other systems, you will find more detailed information in separate articles. The list of articles can be found in the chapter on notification scripts. |
Selecting contacts
The most common procedure is for notifications to be sent to all users who have been registered as a contact for the respective host/service. This is the ‘normal’ and logical procedure, since it is also via the contacts that it is defined which objects each user receives in their GUI display — in effect those objects for which the user is responsible.
You can specify several options when selecting contacts with Add recipient and thus extend the notification to more contacts. Checkmk will automatically delete duplicate contacts. In order for the rule to make sense at least one selection must be made.
The two Restrict previous options to options function somewhat differently.
Here the contacts selected with the other options will again be restricted.
With these you can also create an AND-operator between contact groups, for example, to enable notifications to be sent to all contacts who are members of both the Linux and Data center groups.
By entering Explicit email addresses you can notify persons who are not in fact nominated as users in Checkmk. This of course only makes sense when used in the notification method that actually sends emails.
If, in the chosen method, you have selected Suppress all previous, the notifications will only be deleted for the contact selected here.
|
When using a ticket system, a messenger or an event engine as notification method, you should also observe the notes on these special cases. |
General properties
As with all rules in Checkmk, here you can include a description and a commentary for the rule, or even temporarily deactivate the rule.
With the Allow users to deactivate this notification option you can allow users to 'unsubscribe' from notifications generated by this rule. We show how to do this with the personal notifications.
4.2. Testing notification rules
For testing notification rules, Checkmk offers a smart tool with Test notifications. You can use it to simulate a notification for a host or service and recognize which of your notification rules are effective. In addition to the simulation, you can also have the notification sent.
The quickest way to access the notification test is via Setup > Events > Test notifications. In addition, there are other options for calling from some views in Monitoring (service list and service details) and in the Setup (host properties), in each case in the menu Host > Test notifications.
First, click on one of the two buttons to decide whether the notification is for a Host or a Service. Then select which host or service it should be. As the event you can select a state change or the start of a scheduled downtime. You can add a description of the event to Plugin output. Use the Send out notification for specific method checkbox to specify whether the notification is only simulated or actually sent.
Finally, under Advanced condition simulation there are two further options with which you can define the time and number of notifications. This allows you to test notification rules that only apply during a certain period (e.g. outside of business hours) or that start an escalation after a specified number of repeated notifications.
Click on Test notifications to start the test — and also the dispatch, if you have selected this option. The results are displayed below the Test notifications box:
First comes the summary.
Under Test results you can see how many notification rules apply and how many notifications result from them.
If you have selected Send out notification, the corresponding message notifications have been sent out is displayed here.
This must then immediately lead to an email for this problem.
The line below summarizes the notifications generated from your entries.
By clicking on the 
icon, you can show the notification context.
This allows you to see the environment variables and their values that are valid in the context of this notification.
The following two sections then show more details:
Under Predicted notifications you can see to whom and how notifications would be sent. You will also receive this information about the simulation if you have not selected to send the notification.
Under Global notification rules the notification rules are listed that you have created here.
At this point, only the first column of the table is important, which uses icons to indicate which of the rules apply 
and which do not 
.
|
As usual, you can continue to trigger notifications directly via the GUI as an alternative to testing notifications via simulation, e.g. with the Send custom notification and Fake check results commands. |
4.3. Periodically-repeated notifications and escalation
For some systems, it can make sense not to leave it to a single notification when a problem persists over a longer time frame, for example, for hosts whose host tag Criticality is set to Business critical.
Set up periodically-repeated notifications
Checkmk can be set up so that successive notifications are issued at fixed intervals, until either the problem has been 
acknowledged or solved.
The setting for this can be found in the Periodic notifications during host problems, or respectively, the Periodic notifications during service problems rule set:
Once this option is active, for a persistent problem, Checkmk will trigger regular notifications at the configured intervals. These notifications will receive an incrementing number beginning with 1 (for the initial notification).
Periodic notifications are not only useful for reminding about a problem (and annoying the operator), they also provide a basis for escalations — meaning that after a defined time a notification can be escalated to other recipients.
Set up escalations and understand them
To set up an escalation, create an additional notification rule which uses the Restrict to notification number condition.
If you enter 3 to 99999 as the range for the sequential number, this rule takes effect from the third notification. The escalation can then be performed either by selecting another method, (e.g., SMS), or it can notify other persons (contact selection).
With the Throttling of 'Periodic notifications' option, after a given time the rate of notification repetition can be reduced so that, for example, in the beginning an email can be sent every hour, and later this can be reduced to one email per day.
With multiple notification rules, you can build an escalation model. But how will this escalation then function in practice? Who is notified and when? Here is an example, implemented with one rule for periodically repeated notifications as well as three notification rules: For example:
In the event of a problem being detected in a service, a notification in the form of an email will be triggered every 60 minutes until the problem is either resolved or acknowledged.
Notifications one to five go to the two people responsible for the service.
Notifications six to ten are also sent to the relevant team leader.
From notification eleven on, a daily mail goes to the company management instead.
At 9 o’clock in the morning, a problem occurs at the facility. The two responsible employees are notified of the problem but do not respond (for whatever reason). So at 10, 11, 12 and at 1 p.m. they each receive new emails. From the sixth notification at 2 p.m., the team leader now also receives an email — nevertheless, the problem still does not change. At 3, 4, 5 and 6 p.m., further emails are sent to the team members and the team leader.
At 7 p.m., the third escalation level takes effect: From now on, no more emails are sent to the team members or the team leader. Instead, the company management now receives an email every day at 7 p.m. until the problem is resolved.
As soon as the problem has been fixed and the service in Checkmk goes back to OK, an 'all clear' is automatically sent to the last notified group of people: So in the example above, if the problem is fixed before 2 p.m., to the two team members; if the problem is fixed between 2 and 7 p.m., to the team members and the team leader; and after 7 p.m., only to the company management.
4.4. Deleting notifications by rules
As already mentioned in the selection of the notification method, you will also find the selection option Suppress all previous. In order to be able to understand the functioning of such a rule, it is best to visualize the notification table. Assuming the processing of the rules for a concrete monitoring event is partly complete, and that due to a number of rules the following three notifications have been triggered:
| Who (contact) | How (method) |
|---|---|
Harry Hirsch |
|
Bruno Weizenkeim |
|
Bruno Weizenkeim |
SMS |
Now comes a next rule with the SMS method and the Cancel previous notifications selection. The contact selection chooses the 'Windows' group, in which Bruno Weizenkeim is a member. As a result of this rule, the entry 'Bruno Weizenkeim / SMS' is removed from the table, which then looks like this:
| Who (contact) | How (method) |
|---|---|
Harry Hirsch |
|
Bruno Weizenkeim |
Should a subsequent rule again define an SMS notification for Bruno, then this rule will have priority and the SMS will be added anew to the table.
To summarize:
Rules can suppress (delete) specific notifications.
Deletion rules must come after the rules that create the notifications.
A deletion rule does not actually ‘delete’ a preceding rule, rather it suppresses the notifications that are generated by (possibly multiple) preceding rules.
Subsequent rules can reinstate the previously suppressed notifications.
4.5. Synchronous delivery for HTML emails
You can select and configure the traceable delivery via SMTP for the notification method HTML email by entering the smarthost (with name and port number) and the access data and encryption method:
In the history of the service concerned, you can then track the delivery exactly. Here is an example in which a service — for testing purposes — was manually set to CRIT. The screenshot below shows the notifications for this service, which you can display on the service details page with Service > Service Notifications:
Here you will see the four individual steps in the chronological sequence from bottom to top, as we have already presented them in the chapter on notification history in the in article on the notification basics.
The important difference is that now you can see in the 
topmost entry that the email was successfully delivered to the smarthost and its response is success.
You can also follow the individual steps in the ~/var/log/notify.log file.
The following lines belong to the last step and contain the response of the SMTP server:
2021-08-26 10:02:22,016 [20] [cmk.base.notify] Got spool file d3b417a5 (mysrv;CPU load) for local delivery via mail
2021-08-26 10:02:22,017 [20] [cmk.base.notify] executing /omd/sites/mysite/share/check_mk/notifications/mail
2021-08-26 10:02:29,538 [20] [cmk.base.notify] Output: success 250 - b'2.0.0 Ok: queued as 1BE667EE7D6'The Message-ID 1BE667EE7D6 will appear in the smarthost’s log file.
There — if you are concerned — you can investigate where the email has gotten to.
In any case you can prove that, and when, the email was correctly sent from Checkmk.
Let us repeat the test from above, but this time with a falsely-configured password for the SMTP transfer to the smarthost.
Here you can see in plain text the SMTP error message Error: authentication failed from the smarthost:
What can be done about failed notifications? Again, notifying by email is apparently not a good solution. Instead Checkmk shows a clear warning with red background color in the Overview snap-in:
Here you can:
Click on the text … failed notifications for a list of the failed deliveries.
Click on the
icon to confirm these messages and remove the notice by clicking on 
Confirm in the overview that opens.
Important: Note that direct delivery per SMTP in error situations can lead to a notification script running for a very long time and lead to a timeout. For this reason you are strongly advised to use the notification spooler and to select an asynchronous delivery of notifications.
The conduct with repeatable errors (such as an SMTP timeout) can be defined with Global settings > Notifications > Notification spooler configuration per notification method:
Alongside an optional timeout (the default is 1 minute) and a maximum number of retries, it can also be defined whether the script is permitted to run multiply in parallel and thus send multiple notifications (Maximum concurrent executions). If the notification script is very slow a parallel execution can make sense — however the script must be so-programmed that multiple executions run cleanly (and, for example, that the script doesn’t reserve certain data for itself).
A multiple, parallel delivery over SMTP is unproblematic since the target server can manage multiple parallel connections. This is certainly not the case when delivering directly from SMS via a modem without an additional spooler, and here one should stick with the setting 1.
Important: Traceable delivery via SMTP is not available for bulk notifications!
5. Bulk notifications
5.1. Overview
Everyone who works with monitoring has experienced an isolated problem setting off a veritable flood of (successive) notifications. The principle of the parent hosts is a way of reducing these under specific circumstances, but unfortunately it doesn’t help in all cases.
You can take an example from the Checkmk project itself: Once each day we build Checkmk installation packages for every supported Linux distribution. Our own Checkmk monitoring is set up so that we have a service that is then only OK if the right number of packages have been correctly constructed. It can occasionally happen that a general error in the software hampers the packaging, causing 43 services to go into a CRIT state simultaneously.
We have configured the notifications in such a way that in such a case only a single email listing all 43 notifications in sequence will be sent. This is naturally clearer than 43 single emails, and it also reduces the risk that ‘in the heat of the battle’ one misses a 44th email belonging to quite another problem.
The mode of operation of this bulk notification is very simple. When a notification occurs, at first it will be held back for a short time. Subsequent notifications that occur during this time will be immediately added into the same email. This collecting can be defined for each rule. So, for example, during the day you can operate with individual emails, but overnight with a bulk notification. If a bulk notification is activated you will generally be offered the following options:
The waiting time can be configured as desired. In many cases one minute suffices as by then at the latest all related problems should have appeared. You can of course set a longer time, but that will result in a fundamental delay to the notifications.
Since it naturally makes no sense to throw everything into a single pot, you can specify which groups of problems should be notified collectively. The Host option is very commonly used — this ensures that only notifications from the same host are bundled.
Here are a few additional facts about bulk notifications:
If the bundling is activated in a rule, the activation can be deactivated by a subsequent rule – and vice versa.
The bulk notification always takes place per contact. Each contact has their own 'private collection pot' in effect.
You can limit the size of the pot (Max. notifications per bulk). Once the maximum is reached the bulk notification will immediately be sent.
5.2. Bulk notifications and time periods
What happens when a notification is within the notification period, but the bulk notification that contains this notification — and which comes somewhat later — is outside the notification period? The reverse situation is also possible…
Here a very simple principle applies: all configurations that restrict notifications to time periods are valid only for the actual notification. The subsequent bulk notification will always be delivered independently of all time periods.
6. What happens if no rule is applicable?
One who configures can also make errors. One possible error in the notification configuration could be that a critical monitoring problem is discovered, but not a single notification rule takes effect.
To protect you from such a case, Checkmk offers the setting Fallback email address for notifications. This can be found under Setup > General > Global settings in the Notifications section. Enter an email address here. This email address will then receive notifications for which no notification rule applies.
|
Alternatively, you can also make a user the recipient in his personal settings. The email address stored with the user is used as the fallback address. |
The fallback address will however only be used if no rule applies, not when no notification has been triggered! As we showed in in the section on deletion of notifications, the explicit suppression of notifications is desired — it is not a configuration error.
The entry of a fallback address will be ‘recommended’ on the Notifications page with an onscreen warning:
If - for whatever reason - you only want to get rid of the warning, but at the same time do not want to receive emails at the fallback address, first enter a fallback address anyhow and then create a new rule as the first rule, which deletes all previous notifications. This rule is ineffective on the notification configuration, as no notifications have been created here yet. But with this you can ensure that at least one rule will always apply, thus allowing this warning to be eliminated.
We explicitly advise against this approach, as you may overlook gaps in your rule chain.