Notifications

In the default Checkmk configuration, a user will receive notifications per email when the following prerequisites have been satisfied:

For the successful sending of emails, your Checkmk server must have a functioning SMTP-server configuration. Depending on your Linux distribution, this could utilize, for example, Postfix, Qmail, Exim, or Nullmailer. The configuration will be implemented with your Linux distribution’s resources.

The configuration is generally confined to registering a ‘smarthost’ (also known as an SMTP relay server) to which all emails will be directed. This will then be your firm’s internal SMTP mail server. As a rule smarthosts don’t require authentication in a LAN — which makes matters simple. In some distributions the smarthost will be queried during the installation. With the Checkmk appliance one can configure the smarthost conveniently via the web interface.

You can test the sending of emails easily with the mail command on the command line. Because there are numerous different implementations for this command under Linux, for standardization Checkmk provides the version from the Heirloom mailx project directly in the site user’s search path (as ~/bin/mail). The corresponding configuration file is ~/etc/mail.rc. The best way to test this is as a site user, since the notification scripts will later run with the same permissions.

The email’s content is read from the standard input, the subject specified with -s, and the recipient’s address simply appended as an argument to the end of the command line:

The email should be delivered without delay. If this doesn’t work, information can be found in the SMTP server’s log file in the /var/log directory (see files and directories).

A user’s email address and contact group are defined in the user administration:

In a freshly-generated Checkmk site, initially there is only the Everything contact group. Members of this group are automatically responsible for all hosts and services, and will be notified of every relevant monitoring event by email.

Note: If your Checkmk installation has been generated with an older version, this group could also possibly be called Everybody. This is however illogical, as this group doesn’t contain every user, rather it contains all hosts. Apart from the differing names the function is the same.

Instead of email or SMS, you can also send notifications to a ticket system (such as Jira or ServiceNow), a messenger (Slack, Mattermost) or an event engine (Event Console). There is a separate notification method for each of these special cases, which can be selected in the notification rule. However, you must note the following two points when creating the rule:

To test the notification you could simply set an OK service to CRIT manually. This can be done with the command Fake check results, which you can find in the service list in the Commands menu. If the command is not visible in the list, please click the button Show more once.

This should immediately trigger an email for this problem. At the next regular check the service should revert to OK, thereby triggering a new notification — this time a recovery.

Note that during these tests, by making frequent changes to its state the service will after a while enter the flapping state. Subsequent state changes will no longer trigger notifications. In the Master control snap-in on the sidebar you can temporarily deactivate the detection of flapping (Flap Detection).

Alternatively, you can also send a Custom notification — also using a command:

This does not alter the status of the affected service. This generated notification is however of a different type and — depending on your notification rules — it can behave differently.

When sending HTML emails, you may wish to add additional information or flexibly define a reply address (Reply to) to a specific contact for any queries. For this, there is the Setup > Services > Service monitoring rules > Parameters for HTML Email rule, and in notification rules the HTML email notification method. With these rules you can add a number of parameters such as response address, additional fields with details or free text formatted as HTML.

Note that in the Add HTML section above table (e.g. title, description…​) field, for security reasons only a small set of HTML tags is allowed. These are:

As usual with all rules in Checkmk, a very finely granulated application is possible, so that you can individualize a detailed set of notifications to hosts and services as required.

Checkmk is configured ‘by default’ so that when a monitoring event occurs a notification email is sent to every contact for the affected host or service. This is certainly initially sensible, but in practice many further requirements arise, for example:

Checkmk provides you with maximum flexibility in implementing such requirements via its rule-based mechanism. You enter the configuration with Setup > Events > Notifications.

Note: When you call up the Notification configuration page for the first time, you will see a warning about the non-configured "fallback email address". You can ignore this warning for the moment. We will go into this below.

In the notification configuration, you manage the chain of notification rules, which determine who should be notified and how. When any monitoring event occurs this rule chain will be run through from top to bottom. Each rule has a condition that decides whether the rule actually applies to the situation in question.

If the condition is satisfied the rule determines two things:

Important: In contrast to the rules for hosts and services, here the evaluation also continues after the applicable rule has been satisfied. Subsequent rules can add further notifications. Notifications generated by preceding rules can also be deleted.

The end result of the rule evaluation will be table with a structure something like this:

Now, for each entry in this table the notification script which actually executes the user notification appropriate to the method will be invoked.

If you have a freshly-installed Checkmk, precisely one rule will have been predefined:

This rule defines the above-described default behavior. It is structured as follows:

As usual, you can edit the rule , clone (copy) it , delete it , or create a new rule. Once you have more than a single rule, you can change their processing order by dragging and dropping them with the icon.

Note: Changes to notification rules do not require activation of the changes, rather they take effect immediately.

In the following, we present the general structure of the notification rules with the definitions of general properties, methods, contacts and conditions.

As already mentioned in the selection of the notification method, you will also find the selection option Cancel previous notifications. In order to be able to understand the functioning of such a rule, it is best to visualize the notification table. Assuming the processing of the rules for a concrete monitoring event is partly complete, and that due to a number of rules the following three notifications have been triggered:

Now comes a next rule with the SMS method and the Cancel previous notifications selection. The contact selection chooses the 'Windows' group, in which Bruno Weizenkeim is a member. As a result of this rule, the entry 'Bruno Weizenkeim / SMS' is removed from the table, which then looks like this:

Should a subsequent rule again define an SMS notification for Bruno, then this rule will have priority and the SMS will be added anew to the table.

To summarize:

One who configures can also make errors. One possible error in the notification configuration could be that a critical monitoring problem is discovered, but not a single notification rule takes effect.

To protect you from such a case, Checkmk offers the setting Fallback email address for notifications. This can be found under Setup > General > Global settings in the section Notifications. Enter an email address here. This email address will then receive notifications for which no notification rule applies.

Note: Alternatively, you can also make a user the recipient in his personal settings. The email address stored with the user is used as the fallback address.

The fallback address will however only be used if no rule applies, not when no notification has been triggered! As we showed in the previous chapter, the explicit suppression of notifications is desired — it is not a configuration error.

The entry of a fallback address will be ‘recommended’ on the Notification configuration page with an onscreen warning:

If - for whatever reason - you only want to get rid of the warning, but at the same time do not want to receive emails at the fallback address, first enter a fallback address anyhow and then create a new rule as the first rule, which deletes all previous notifications. This rule is ineffective on the notification configuration, as no notifications have been created here yet. But with this you can ensure that at least one rule will always apply, thus allowing this warning to be eliminated.

We explicitly advise against this approach, as you may overlook gaps in your rule chain.

A useful feature in Checkmk’s notification system is the one with which users — even without administrator rights — can customize notifications. Users can:

The entry point from the user’s point of view is the User menu, and there the entry Notification rules. On the Your personal notification rules page, a new rule can be created with Add rule.

Custom rules are structured like the normal rules — with one difference: they do not contain a contact selection. The user himself is automatically selected as the contact. This means that a user can only add or delete notifications for himself.

However, the user can only delete notifications if the allow users to deactivate this notification option is activated in the rule that creates them:

In the order of notification rules, the custom rules always come after the global rules and so can adjust the notification table that has been generated so far. So, except for the deletion blocking just described, the global rules always apply as the default setting that can be customized by the user.

If you want to prevent customization altogether, you can revoke the permission General Permissions > Edit personal notification settings of the user role.

As an administrator, you can display all user rules if in the menu you select Display > Show user rules:

After the global rules, the user rules are listed, which you can also edit with .

The complete disabling of notifications by a user is protected with the permission General Permissions > Disable all personal notifications, which is set to no for the user role user by default. A user will only see the corresponding checkboxes in his personal settings if you explicitly assign this right to the user role:

As an administrator with access to the user’s personal settings, you can carry out disablement actions on the user’s behalf — even if the permission as described above is not present. You can find this setting under Setup > Users > Users and then in the properties of the user profile. With this, for example, you can very quickly silence a holidaying colleague’s notifications — without needing to alter the actual configuration.

A large part of the Checkmk notification system’s complexity is due to its numerous tuning options, with which unimportant notifications can be avoided. Most of these will be situations in which notifications are already being delayed or suppressed when they occur. Additionally, the monitoring core has a built-in intelligence that suppresses certain notifications by default. We would like to address all of these aspects in this chapter.

When a host or service is in a scheduled downtime the object’s notifications will be suppressed. This is – alongside a correct evaluation of availabilities — the most important reason for the actual provision of downtimes in monitoring. The following details are relevant to this:

You can define a notification period for each host and service during configuration. This is a time period which defines the time frame within which the notification should be constrained.

The configuration is performed using the Monitoring Configuration > Notification period for hosts, or respectively the Notification period for services rule set, which you can quickly find via the search in Setup menu. An object that is not currently in a notification period will be flagged with a gray pause icon .

Monitoring events for an object that is not currently in its notification period will not be notified. Such notifications will be ‘reissued’ when the notification period is again active – if the host/service is still in a problem state. Only the latest state will be notified even if multiple changes to the object’s state have occurred during the time outside the notification period.

Incidentally, in the notification rules it is also possible to restrict a notification to a specific time period. In this way you can additionally restrict the time ranges. However, notifications that have been discarded due to a rule with time conditions will not automatically be repeated later!

If a host has completely failed, or is at least inaccessible to the monitoring, then obviously its services can no longer be monitored. Active checks will then as a rule register CRIT or UNKNOWN, since these will be actively attempting to access the host and will thereby run into an error. In such a situation all other checks — thus the great majority — will be omitted and will thus remain in their old state. These will be flagged with the stale time icon .

It would naturally be very cumbersome if all active checks in such a state were to notify their problems. For example, if a web server is not reachable – and this has already been notified – it would not be very helpful to additionally generate an email for every single one of its dependent HTTP services.

To minimize such situations, as a basic principle the monitoring core only generates notifications for services if the host is in the UP state. This is also the reason why host accessibility is separately verified. If not otherwise configured, this verification will be achieved with a Smart Ping or ping.

If you are using the Raw Edition (or one of the commercial editions with a Nagios core), in isolated cases it can nonetheless occur that a host problem generates a notification for an active service. The reason for this is that Nagios regards the results of host checks as still being valid for a short time into the future. If even only a few seconds have elapsed between the last successful ping to the server and the next active check, Nagios can still assess the host as UP even though it is in fact DOWN. In contrast, the Checkmk Micro Core (CMC) will hold the service notification in a ‘standby’ mode until the host state has been verified, thus reliably minimizing undesired notifications.

Imagine that an important network router to a company location with hundreds of hosts fails. All of its hosts will then be unavailable to the monitoring and become DOWN. Hundreds of notifications will therefore be triggered. Not good.

In order to avoid such problems the router can be defined as a parent host for its hosts. If there are redundant hosts, multiple parents can also be defined. As soon as all parents enter a DOWN state, the hosts that are no longer reachable will be flagged with the UNREACH state and their notifications will be suppressed. The problem with the router itself will of course still be notified.

By the way, the CMC operates internally in a slightly different manner to Nagios. In order to reduce false alarms, but still process genuine notifications, it pays very close attention to the exact times of the relevant host checks. If a host check fails the core will wait for the result of the host check on the parent host before generating a notification. This wait is asynchronous and has no effect on the general monitoring. Notifications from hosts can thereby be subject to minimal delays.

With the Enable/disable notifications for hosts, or respectively, the Enable/disable notifications for services rule sets you can specify hosts and services for which generally no notifications are to be issued. As mentioned above the core then suppresses notifications. A subsequent notification rule that ‘subscribes’ to notifications for such services will be ineffective, as the notifications are simply not generated.

It is also possible to temporarily disable notifications for individual hosts or services via a command.

However, this requires that the permission Commands on host and services > Enable/disable notifications is assigned to the user role. By default, this is not the case for any role.

With the assigned permission, you can disable (and later enable) notifications from hosts and services with the Commands > Notifications command:

Such hosts or services will then be marked with an icon.

Since commands — in contrast to rules — require neither configuration permissions nor an activate changes, they can be a quick workaround for reacting promptly to a situation.

Important: In contrast to scheduled downtimes, disabled notifications have no influence on the availability evaluations. If during an unplanned outage you really only want to disable the notifications without wishing to distort the availability statistics, you should not register a scheduled downtime!

In the Master control snap-in in the sidebar you will find a master switch for Notifications:

This switch is incredibly useful if you plan to make bigger system changes, during which an error could under the circumstances force many services into a CRIT state. You can use the switch to avoid upsetting your colleagues with a flood of useless emails. Remember to re-enable the notifications when you are finished.

Each site in a distributed monitoring has one of these switches. Switching off the central site’s notifications still allows remote sites to activate notifications — even though these are directed to and delivered from the central site.

Important: Notifications that would have been triggered during the time when notifications were disabled will not be repeated later when they are re-enabled.

You may possibly have services that occasionally enter a problem state for short periods, but the stops are very brief and are not critical for you. In such cases notifications are very annoying, but are easily suppressed. The Delay host notifications and Delay service notifications rule sets serve this situation.

You specify a time in minutes here — and a notification will be delayed until this time has expired. Should the OK / UP state occurs again before then, no notification will be triggered. Naturally this also means that the notification of a genuine problem will be delayed.

Obviously even better than delaying notifications would be the elimination of the actual cause of the sporadic problems — but that is of course another story…​

Another very similar method for delaying notifications is to allow multiple check attempts when a service enters a problem state. This is achieved with the Maximum number of check attempts for hosts, or respectively, the Maximum number of check attempts for service rule set.

If you set a value of 3 here, for example, a check with a CRIT result will at first not trigger a notification. This is referred to as a CRIT soft state. The hard state remains OK. Only if three successive attempts return a not-OK-state will the service switch to the hard state and a notification be triggered.

In contrast to delayed notifications, here you have the option of defining views so that such problems are not displayed. A BI aggregate can also be constructed so that only hard states are included — not soft ones.

When a host or service frequently changes its state over a short time it is regarded as flapping. This is an actual state. The principle here is the reduction of excessive notifications during phases when a service is not (quite) running stably. Such phases can also be specially evaluated in the availability statistics.

Flapping objects are marked with the icon. As long as an object is flapping, successive state changes trigger no further notifications. A notification will however be triggered whenever the object enters or leaves the flapping state.

The system’s recognition of flapping can be influenced in the following ways:

Show the context sensitive help with Help > Show inline help for details on the customizable values.

For some systems, it can make sense not to leave it to a single notification when a problem persists over a longer time frame, for example, for hosts whose host tag Criticality is set to Business critical.

To get started, we will show you how to view the history of notifications at the host and service level in Checkmk to be able to track the notification process.

A monitoring event that causes Checkmk to trigger a notification is, for example, the change of state of a service. You can manually trigger this state change with the command Fake check results for testing purposes, as we have already shown in the chapter on the testing of a simple notification by email.

For a notification test, you can move a service from the OK state to CRIT in this way. If you now display the notifications for this service on the service details page with Service > Service Notifications, you will see the following entries:

The most recent entry is at the top of the list. However, the first entry is at the bottom, so let’s look at the individual entries from bottom to top:

To help in the correct understanding of the contexts for all of the various setting options and basic conditions, and to enable an accurate problem diagnosis when a notification appears or does not appear as expected, here we will describe all of the details of the notification process including all of the components involved.

Note: The notification history that we have shown above for a service can also be displayed for a host: on the host details page in the Host menu for the host itself (Notifications of host menu item) and also for the host with all its services (Notifications of host & services).

The following components are involved in the Checkmk notification system:

Once the core has generated a raw notification, this runs through the chain of notification rules – resulting in a table of notifications. Alongside the data from the raw notification, every notification contains the following additional information:

In a synchronous delivery, for every entry in the table an appropriate notification script will now be executed. In an asynchronous delivery a notification will be passed as a file to the notification spooler.

A powerful supplementary function of the commercial editions is the notification spooler. This enables an asynchronous delivery of notifications. What does asynchronous mean in this context?

A synchronous delivery is then feasible if the notification script runs quickly, and above all can’t lead to some sort of timeout. With notification methods that access existing spoolers that is a given. Spool services from the system can be used particularly with email and SMS. The notification script passes a file to the spooler — with this procedure no wait state can occur.

When using the traceable delivery via SMTP or other scripts which establish network connections, you should always employ asynchronous delivery. This also applies to scripts that send text messages (SMS) via HTTP over the internet. The timeouts when building a connection to a network service can take up to several minutes, causing a jam as described above.

The good news is that asynchronous delivery is enabled by default in Checkmk. For one thing, the notification spooler (mknotifyd) is also started when the site is started, which you can check with the following command:

On the other hand, asynchronous delivery (Asynchronous local delivery by notification spooler) is selected in Global settings > Notifications > Notification Spooling:

Everyone who works with monitoring has experienced an isolated problem setting off a veritable flood of (successive) notifications. The principle of the parent hosts is a way of reducing these under specific circumstances, but unfortunately it doesn’t help in all cases.

You can take an example from the Checkmk project itself: Once each day we build Checkmk installation packages for every supported Linux distribution. Our own Checkmk monitoring is set up so that we have a service that is then only OK if the right number of packages have been correctly constructed. It can occasionally happen that a general error in the software hampers the packaging, causing 43 services to go into a CRIT state simultaneously.

We have configured the notifications in such a way that in such a case only a single email listing all 43 notifications in sequence will be sent. This is naturally clearer than 43 single emails, and it also reduces the risk that ‘in the heat of the battle’ one misses a 44th email belonging to quite another problem.

The mode of operation of this bulk notification is very simple. When a notification occurs, at first it will be held back for a short time. Subsequent notifications that occur during this time will be immediately added into the same email. This collecting can be defined for each rule. So, for example, during the day you can operate with individual emails, but overnight with a bulk notification. If a bulk notification is activated you will generally be offered the following options:

The waiting time can be configured as desired. In many cases one minute suffices as by then at the latest all related problems should have appeared. You can of course set a longer time, but that will result in a fundamental delay to the notifications.

Since it naturally makes no sense to throw everything into a single pot, you can specify which groups of problems should be notified collectively. The Host option is very commonly used — this ensures that only notifications from the same host are bundled.

Here are a few additional facts about bulk notifications:

What happens when a notification is within the notification period, but the bulk notification that contains it — and which comes somewhat later — is outside the notification period? The reverse situation is also possible…​

Here a very simple principle applies: all configurations that restrict notifications to time periods are valid only for the actual notification. The subsequent bulk notification will always be delivered independently of all time periods.

Monitoring is only useful when one can rely on it. This requires that notifications are received reliably and promptly. Unfortunately email delivery is not completely ideal however. The dispatch is usually processed by passing the email to the local SMTP server. This attempts to deliver the email autonomously and asynchronously.

With a temporary error (e.g., a case where the receiving SMTP server is not reachable) the email will be put into a queue and a later a new attempt will be made. This ‘later’ will as a rule be after 15-30 minutes. By then the notification could be far too late!

If the email really can’t be delivered the SMTP server creates a nice error message in its log file and attempts to generate an error email to the ‘sender’. But the monitoring system is not a real sender and also cannot receive emails. It follows that such errors simply disappear and notifications are then absent.

Important: Traceable notifications are not available for bulk notifications!

The commercial editions provide the possibility of a traceable delivery via SMTP. This it intentionally does without the help of the local mail server. Instead Checkmk itself sends the email to your smarthost via SMTP, and then it evaluates the SMTP response itself.

In this way, not only are SMTP errors treated intelligently, but a correct delivery is also precisely documented. It is a bit like a registered letter: Checkmk receives a receipt from the SMTP smarthost (receiving server) verifying that the email has been accepted — including a mail ID.

You can select and configure the traceable delivery via SMTP for the notification method HTML email by entering the smarthost (with name and port number) and the access data and encryption method:

In the history of the service concerned, you can then track the delivery exactly. Here is an example in which a service — for testing purposes — was manually set to CRIT. The screenshot below shows the notifications for this service, which you can display on the service details page with Service > Service Notifications:

Here you will see the four individual steps in the chronological sequence from bottom to top, as we have already presented them in the chapter on notification history. The important difference is that now you can see in the topmost entry that the email was successfully delivered to the smarthost and its response is success.

You can also follow the individual steps in the notify.log file. The following lines belong to the last step and contain the response of the SMTP server:

The Message-ID 1BE667EE7D6 will appear in the smarthost’s log file. There — if you are concerned — you can investigate where the email has gotten to. In any case you can prove that, and when, the email was correctly sent from Checkmk.

Let us repeat the test from above, but this time with a falsely-configured password for the SMTP transfer to the smarthost. Here you can see in plain text the SMTP error message Error: authentication failed from the smarthost:

What can be done about failed notifications? Again, notifying by email is apparently not a good solution. Instead Checkmk shows a clear warning with red background color in the Overview:

Here you can:

Important: Note that direct delivery per SMTP in error situations can lead to a notification script running for a very long time and lead to a timeout. For this reason you are strongly advised to use the notification spooler and to select an asynchronous delivery of notifications.

The conduct with repeatable errors (such as an SMTP timeout) can be defined with Global settings > Notifications > Notification spooler configuration per notification method:

Alongside an optional timeout (the default is 1 minute) and a maximum number of retries, it can also be defined whether the script is permitted to run multiply in parallel and thus send multiple notifications (Maximum concurrent executions). If the notification script is very slow a parallel execution can make sense — however the script must be so-programmed that multiple executions run cleanly (and, for example, that the script doesn’t reserve certain data for itself).

A multiple, parallel delivery over SMTP is unproblematic since the target server can manage multiple parallel connections. This is certainly not the case when delivering directly from SMS via a modem without an additional spooler, and here one should stick with the setting 1.

A synchronous delivery including error messages and traceability has to date only been implemented for HTML emails. How one can return an error status in a self-written notification script can be found in the chapter on writing your own scripts.

Notification can occur in manifold and individual ways. Typical examples are:

For this reason Checkmk provides a very simple interface which enables you to write your own notification scripts. These can be written in any Linux-supported programming language — even though Shell, Perl and Python together have 95 % of the ‘market’.

The standard scripts included with Checkmk can be found in ~/share/check_mk/notifications. This directory is a component of the software and is not intended to be changed. Instead, save your own scripts in ~/local/share/check_mk/notifications. Ensure that your scripts are executable (chmod +x). They will then be found automatically and made available for selection in the notification rules.

Should you wish to customize a standard script, simply copy it from ~/share/check_mk/notifications to ~/local/share/check_mk/notifications and there make your changes in the copy. If you retain the original name, your script will be substituted automatically for the standard version and no changes will need to be made to the existing notification rules.

Some more sample scripts are included with the software in ~/share/doc/check_mk/treasures/notifications. You can use these as templates for customization. The configuration will generally take place directly in the script — tips covering this can be found there in the comments.

In the case of a notification your script will be called up with the site user’s permissions. In environment variables, (those that begin with NOTIFY_), it will receive all of the information about the affected host/service, the monitoring event, the contacts to be notified, and the parameters specified in the notification rule.

Texts that the script writes to the standard output (with print, echo, etc.), appear in the notification module’s log file ~/var/log/notify.log.

Notification scripts have the option of using an exit code to communicate whether a replicable or final error has occurred:

Additionally, in all cases the standard output from the notification script, together with the status will be entered into the host’s/service’s notification history and will therefore be visible in the GUI.

Important: Traceable notifications are not available for bulk notifications!

The treatment of notification errors from the user’s point of view will be explained in the chapter on traceable delivery via SMTP.

As an example you can create a script that writes all of the information about the notification to a file. The coding language is the Bash Linux shell:

Then make the script executable:

Here are a couple of explanations concerning the script:

So that the script will be used you must define it as a method in a notification rule. Self-written scripts have no parameter declaration, therefore all of the checkboxes such as those offered, for example, in the HTML Email method, will be missing. Instead you can enter a list of texts as parameters that can be available as NOTIFY_PARAMETER_1, NOTIFY_PARAMETER_2, etc, to the script. For a test provide the parameters Fröhn, Klabuster and Feinbein:

Now to test, set the service CPU load on the host myserver to CRIT — with the command Fake check results as described above. In the log file of the notification module ~/var/log/notify.log you then see the execution of the script including parameters and the generated spool file.:

The file ~/tmp/foobar.out will now contain an alphabetic list of all Checkmk environment variables that include information concerning the notification. Here you can orient yourself with which values are available to your script. Here are the first ten lines:

The parameters can also be found:

In the above example you have seen a number of environment variables that will be passed to the script. Precisely which variables will be available will depend on the type of notification and also on the Checkmk version and edition being used. Alongside the trick with the env command there are two further ways of getting a complete list of all variables:

Below is a list of the most important variables:

If your script should support bulk notifications, it will need to be specially prepared, since the script must deliver multiple notifications simultaneously. For this reason a delivery using environment variables also doesn’t function practicably.

Give your script a name in the third line in the header as below — the notification module will then send the notifications through the standard input:

Through the standard input the script will receive blocks of variables. Each line has the form: NAME=VALUE. Blocks are separated by blank lines. The ASCII character with the code 1 (\a) is used to represent newlines within the text.

The first block contains a list of general variables (e.g., call parameters). Each subsequent block assembles the variables into a notification.

The best recommendation is to try it yourself with a simple test that writes the complete data to a file so that you can see how the data is sent. You can use the following notification script for this purpose:

Test the script as described above and additionally activate the Notification Bulking in the notification rule.

As delivered, Checkmk already provides a whole range of scripts for connecting to popular and widely used instant messaging services, incident management platforms and ticket systems. You can find out how to use these scripts in the following articles:

The SMTP server’s log files are system files and their absolute paths are listed here below. Precisely where the log files are stored will depend on your Linux distribution.