Business Intelligence (BI)

Before you begin step by step with the practical application of BI, you first need know a few terms:

Each application formalized with BI is called an Aggregation, since a general state is aggregated from many individual states.

An aggregation is constructed as a 'tree' of objects. These objects are called nodes. The bottom nodes — the leaves of the tree — are the hosts and services in your Checkmk instances. The remaining nodes are artificially-created BI objects.

Each node is created by a rule. This also applies to the root of the tree — the topmost node. These rules determine which nodes hang under another node, and how from their states the state of the higher node is to be determined.

The top node of an aggregation — the root of the tree — is also generated with a rule. In this way a rule can generate multiple aggregations.

The easiest way understand this is to use a concrete example. We have thought up the 'Mystery Application' for this article. Suppose that this is an important application in an unspecified company. Among other things, five servers and two network switches are playing an important role. So that you can better understand the example, we use simple names like srv-mys-1 or switch-1. The following diagram gives a simple overview of the structure:

If you want to work through the following steps one by one, you can simply replicate the monitoring objects as shown in our example. For a test it is sufficient if you clone an existing host several times and name the clones accordingly. Later there are a few services to be added into the game, for which you then have time to record the relevant hosts in the monitoring. Even there you can cheat again: with simple dummy-local-checks you will quickly get matching services to play with.

The hosts will then look something like this in the monitoring:

Start with something simple — with the simplest possible meaningful aggregation — an aggregation with only two nodes. You then want to summarize the states of the hosts switch-1 and switch-2. The aggregation should be called Network and should be OK if both switches are available. In the case of a partial failure, it should go to WARN, and if both switches are off, CRIT.

Get started: configure BI via Setup > Business Intelligence > Business Intelligence. The configuration of the rules and aggregations is performed within the configuration packages — the BI Packs. The packages are not only practical because you can better manage more complex configurations with them — you can also apply permissions to a package and assign certain contact groups — and even allow users without admin rights — permissions to edit parts of the configuration. But more on that later …​

The first time you call the BI module it should something like this:

A package titled Default Pack is already present. This contains a demo for an aggregation which summarizes the data for an individual host.

For this example it is best to create a new package — with the Add BI Pack button — which you name Mystery. As always in Checkmk, specify an internal ID (mystery) which cannot be changed later, and a descriptive title. The Public option is needed by other users if there are rules in this package they want to use for their own rules or aggregations. Because you probably want do your experiments alone in peace, leave this option disabled:

After the creation you will of course find two packages in the main list:

With each entry is a symbol for editing the properties (), and a symbol to get to the actual content of the Package (), which is where you want to go now. Once there, create your first rule right away via Add rule.

As always in Checkmk, this rule also needs to have a unique ID and a title. The title of the rule not only has a documentation function, but will later also be visible as the name of the node this rule creates:

The next box is named Child Node Generation, and is the most important. Here you specify which objects in this node should be summarized. This can either be other BI nodes — for which you would choose a different BI rule — or be monitoring objects, i.e., hosts or services.

For the first example select the second variant (State of a host) and create two objects as children — namely the two hosts switch-1 and switch-2. This is done with the Add child node generator button. Here you naturally choose State of a host, and enter a name for each host:

In the third and final box, Aggregation Function, you specify how the monitoring state of the node should be calculated. The basis for this is always the list of states of the subnodes. Different logical links are possible.

Pre-selected is Best — take best of all node states. That would mean that the node becomes CRIT when all sub-nodes are CRIT or DOWN. As mentioned above this should not be the case here. Choose instead Count the number of nodes in state OK to get the number of subnodes with state OK as a yardstick. Here the numbers 2 and 1 are suggested for the thresholds. That is great because it is exactly what you need:

This is how the filled-out mask will look:

With a click on Create you will have your first rule:

Now it is important to understand that a rule is not yet an aggregation. Checkmk can not yet know if this is everything or just part of a bigger tree! Real BI objects are only created and become visible in the status interface when you create an Aggregation. To do this, switch to the list of aggregations.

The button takes you to a mask for creating a new aggregation. There is little to fill in here. In the Aggregation groups you can specify any name of your choosing. These names then appear in the status interface as groups, under which all of those aggregations which share this group name become visible. This is actually the same concept as with hashtags or keywords.

You define the content of the aggregation via Add new element. Select the setting Call a rule and at Rule: the rule you just created (and before that the rule package it is in).

If you now save the aggregation with , you will be done! Your first aggregation should now appear in the status interface — assuming that you in fact also have at least one of the hosts switch-1 or switch-2!

If you have done everything correctly you will now be able to see your first aggregation in the status interface. The easiest way to do this is via Monitor > Business Intelligence > All Aggregations:

Take a closer look at the appearance of the BI tree. The following example shows your mini-aggregation in a situation where one of the two switches is DOWN, and the other UP. As desired, the aggregation enters the WARN state:

You can also see that in order to standardize hosts and services, the host that is DOWN, is treated almost like a service that is CRIT. Likewise UP accordingly becomes OK.

The leaves of the tree show the states of hosts and services. The host name — and for services also the service name — is clickable and takes you to the current state of the corresponding object. Furthermore, you can also see the last output from the check plug-in.

To the left of each aggregation you will find two symbols: and . With the first icon —  — you come to a page that displays just that single aggregation. This is naturally mainly useful if you have created more than one aggregation. It is for, example, well-suited as a bookmark. will take you to the calculation of the availability. More on this later.

To the left of the host name you will find an interesting icon: . This allows a 'what if?' analysis. The idea behind this is simple: through clicking on the icon it will switch the object to another state as a test — however only for the BI interface — NOT for real! Multiple clicks will take you from (OK) via (WARN), (CRIT) and (UNKNOWN), and back to .

BI then constructs the complete tree based on the assumed state. The following figure shows the minimum aggregation under the assumption that alongside switch-1 which has actually failed, that switch-2 would also be DOWN:

The overall state of the aggregation thereby goes from WARN to CRIT. At the same time the state’s color is backed by a checked pattern. This pattern indicates to you that the real state is actually different. This is not always the case, because some changes in a host or service are no longer relevant to the overall condition — for example, if the one in question is already CRIT.

You can use this 'what if?' analysis in several ways, for example:

In the latter scenario, as a test you set the device to be serviced or its services to . If the whole aggregation then remains OK, it must mean that the failure can currently be compensated for by redundancy.

There is another way to test the BI aggregations: by directly changing the actual state of an object. This is especially practical in a test system.

For this purpose, the commands have a host/service command named Fake check results. It is by default only available for the Administrator role. This method has been used, for example, for the creation of the screenshots used in this article where switch-1 has been set to DOWN. This is where the telltale text Manually set to Down by cmkadmin comes from.

Here is a helpful little hint: If you work with this method, it’s best to disable the active checks for the relevant hosts and services, otherwise at the next check interval they will immediately go back to their actual state. If you are lazy just do it globally via the Master Control sidebar element. Just — NEVER forget to turn it back on afterwards!

While creating the aggregation we briefly addressed the possibilities of the Aggregation Groups input. In the example you simply confirmed the suggested Main here. You are of course completely free in the allocation of names, and you can also assign an aggregation to multiple groups.

Groups become important when the number of aggregations possibly exceeds want you to see on a screen. You get to a group by clicking on one of the displayed group names on the All aggregations page — in our example above that is simply on the Main heading. Of course, if so far you only have this single aggregation not much will change. However if you look closely, you will realize that:

If you want to visit this view more often, simply bookmark it — preferably with the Bookmarks element in the sidebar.

Once you have set up BI aggregations, in the context menu at your hosts and services you will find a new icon:

This icon takes you to the list of all aggregations in which the affected host or service is included.

Now that you have a slightly more interesting tree you can get a little closer to dealing with the various display options that CMK offers. The starting point for these are the Modify display options, which you can access via the Display menu. This opens a box with various options. The content of the box always conforms to the elements shown on the page. In the case of BI you can currently find four options:

Finally, you can set a Refresh interval of 30, 60 or 90 seconds and specify the number of columns via Entries per row.

From version 1.6.0, in addition to tabular representations Checkmk also masters the visualization of BI aggregations. You can view aggregations from a new perspective, and sometimes more clearly. You will find the BI Visualization via in the regular Aggregations View.

You can move the tree freely by clicking on the background, and scale the entire display using the mouse wheel. As soon as the mouse pointer is over an individual node, you get the node’s associated state information via a hover window. Use the mouse wheel to scale the length of the tree’s branches.

Clicking on the leaf nodes takes you directly to the detailed views of the host or service. A right click on the other nodes — depending on the type of node — gives access to display options and, for example, the responsible rule itself — Edit rule in the below image.

Continue with the configuration. Now it’s time to really get down to business. So far the example has been so simple that it was possible to individually list all of the objects in the aggregation without difficulty. But what if things get more complex? What if you want to formulate many recurring same or similar dependencies? What if an application includes not a single, but multiple instances? What if you should want to merge hundreds of a database’s individual services into one BI node?

Well, for such requirements you need more powerful methods of configuration. And these are exactly what distinguishes Checkmk BI over other tools — and unfortunately here the learning curve is a bit steeper. It is also the reason why Checkmk BI does not allow itself to be configured by „drag and drop“. Once you get to know the possibilities however, you will certainly not want to go without them.

Let’s start with the parameters. Take the following situation: you not only want to know if the two switches are UP, but also want know the state of the two ports that are responsible for the uplink. In overall terms, it concerns the following four services:

Now the node Switch 1 & 2 should be extended to replace the two host states for switches 1 and 2 so that each has a subnode showing the host state and the two uplink interfaces. These two subnodes should be Switch 1 or Switch 2.

Actually you now need two new rules — one for each switch. It is better to do this by creating a new <switch> rule, and equip it with a parameter. This parameter is a variable that you call when you call the rule from the parent node — which here can be provided by the old rule Switch 1 & 2. In this example you can simply pass either a 1 or a 2. The parameter gets a name which you can choose freely. Take here for example the name NUMBER. The spelling with capital letters is purely arbitrary, and if you find lowercase letters more beautiful you are also free to use these.

And the rule’s heading will look like this:

You can choose switch as the ID for the new rule. At parameter simply enter the name of the variable: NUMBER. Also important now is that the variable is used in the rule’s Rule Title so that both nodes are not just called switch and thus have the same name. When using the variable a leading and trailing dollar sign is set — as usual at many places in Checkmk. As a result the two nodes will then be called Switch 1 and Switch 2.

The subject of regular expressions is again worth a closer look. When matching the service name we have at the beginning tacitly understated that it basically only concerns regular expressions. As just mentioned, there is a prefix match.

So in a BI node, if for example, under service name you specify disk, all of the of the host in question’s services that begin with Disk will be captured.

The following principles generally apply:

Maybe that sounds a bit bold for you! Is not it dangerous to just silently omit things that should be there if they are missing?

Well — over time you will notice how practical this concept is, because this will allow you to write 'smart' rules that can react to very different situations. Is there a service that does not exist with every instance of an application? No problem — it is only considered if it is there! Or can hosts or services be temporarily removed from monitoring? These then simply disappear from BI without leading to errors or the like. BI is not there to see if your monitoring configuration is complete!

Incidentally — this principle also applies to explicitly defined services, since these do not actually exist because the service names are always viewed as regular expressions even if they do not contain special characters such as .*. It is always automatically a search pattern.

But you can still automate further and, above all, react flexibly to changes. Continue with the example of the two application servers srv-mys-1 and srv-mys-2 from the example. Your tree should continue to grow. The Infrastructure node should slip to level 2. And as a definitive root, there should be a rule with the title The Mystery Application under which everything will hang. Alongside Infrastructure there should be a node named Mystery Servers. Under this the (currently) two mystery-servers are supposed to hang. In each a few generic services come into the aggregation. The result should look like this:

Similar to the host search, there is also a child generator type called Create nodes based on a service search. Here is an example:

You can use () here – bracketing partial expressions – both at the host and at the service, where:

And never forget that you can always use to get inline help.

In your attempts you may have stumbled over the child generator State of remaining services. This generates a node for any of your host’s services that have not yet been sorted into your BI aggregation. This is useful if you use BI to combine the states of all of a host’s services into clearly-arranged groups - as it is is done in the included example.

Again, back to the rule packages. For all editing actions in BI you usually need to have the Administrator role. More precisely, for BI there are two permissions, to find under Setup > Users > Roles & permissions:

By default the User role is only the first of the two active permissions. Normal users can only work in such rule packages for which they have been defined as a contact. This is done in the Details of the rule package.

In the following example Permitted Contact Groups the The Mystery Admins contact group has been authorized — thus all members of this group can now edit the rules in this package:

By the way, with Public > Allow all users to refer to rules contained in this pack you can allow other users to at least use the rules contained here — i.e. to (elsewhere) define their own rules — which can then invoke these rules as subnodes.

How is it with the actual visibility of the aggregations in the Status Interface? Which contacts are allowed to see something?

Well, you cannot assign any rights in the BI aggregations themselves. This is performed indirectly through the visibility of the hosts and services, and it is governed by the See all hosts and services option under Setup > Roles & Permissions:

In the User role, this right is by default disabled. Normal users can see only shared hosts and services, and in BI these are expressed in such a way that they can see exactly all of the BI aggregations which contain at least one shared host or service. Such aggregations however contain only these authorized objects, and they may therefore be somewhat 'thinned out'. And this in turn means that they can have different states for different users!

Whether that is good or bad depends on what you want. If in doubt you can toggle the permission, and through a detour via BI allow some or all users to see hosts and services for which they are not contacts — and thus ensure that the state of an aggregation is always the same for everyone.

Of course this whole issue only matters if there are in fact aggregations that are so colorfully thrown together that only some users are contacts only for parts of it.

How does BI actually manage maintenance times? Well, we have thought long and hard about the matter, and discussed it with many users — the result is as follows:

To understand which rule BI calculates the „in maintenance“ state, it helps when you are reminded of what the real idea behind maintenance times is: The object in question is currently being worked on. Failures can be expected. Even if the object is currently OK, you should not rely on it. It can become CRIT at any time. This is known and documented — it should therefore not trigger a notification.

This idea can be transferred 1:1 into BI: In the aggregation there may be a few hosts and services that are currently in maintenance. Whether these are just OK or CRIT does not play a role, because it is actually a coincidence if during the maintenance work the objects sometimes go off and on again, or not. Just because there is a maintenance object in the unit it does not immediately mean that the application that maps the aggregation is itself 'threatened' and must also be marked as „in maintenance“. It can also have an installed redundancy which compensates for the failure of the objects in maintenance. Only if such a failure would actually lead to a CRIT state for the aggregation — so there is not not enough redundancy and the aggregation really is threatened — only then will Checkmk mark it as 'in maintenance'. Where here as well the current state of the objects generally does not matter.

To put it more concisely, the exact rule is as follows:

If a CRIT state of a host/service would result in a CRIT state of the aggregation, an „in maintenance“ state of that host/service results in an „in maintenance“ state of the aggregation.

Important: the real current state of the hosts/services plays no role in the calculation - what is in maintenance is assumed to be CRIT in the BI logic. Why? Because an UP or OK state during a maintenance period is pure coincidence, for example if a host reports UP for a few seconds in between several restarts.

And here we have another example. To save space, this is a variant with only one mystery server instead of two:

First, the host switch-1 is under maintenance. For the Infrastructure node this has no effect, because switch-2 is not in maintenance, and thus Infrastructure is also not in maintenance. There is therefore no icon for derived maintenance times.

But, the service Memory on srv-mys-1 is also under maintenance. This one is not redundant. The maintenance is therefore inherited by the father node Mystery Server 1, then continues up to Mystery Servers and finally to the top node The Mystery Application. So this top node is also in maintenance.

We wrote above that you cannot manually put a BI aggregation into maintenance time? That’s only half true, since in fact you can find a command for setting maintenance times in BI aggregations! But this does nothing more than to record a maintenance entry for each host and service in the aggregation! This of course usually leads to the aggregation itself being flagged as in maintenance. But that is only indirect.

Above you have seen that the maintenance time calculation is based on an assumed CRIT state. In the properties of an aggregation you can customize the algorithm so that a node that assumes the WARN state is marked as in maintenance. The option for this is called Escalate downtimes based on aggregated WARN state:

The basic assumption remains that objects under maintenance are CRIT. There is only a difference where, due to the aggregation function in which a CRIT can become a WARN — as was the case in our very first example with Count the number of nodes in state OK. Here a maintenance time would already have been be accepted if only one of the two switches was in maintenance.

Quite similar to the process with the maintenance times is that if a problem has been acknowledged the information is also calculated automatically by BI. This time the state of the objects certainly plays a role.

The idea here is to transfer the following concept to BI: An object has a problem (WARN, CRIT), but this is known, and someone is working on it ().

You can calculate this for an aggregation as follows:

However if the aggregation were to remain WARN or CRIT, then it would not be considered as acknowledged, because then there must be at least one important problem that has not been acknowledged, and thus the OK state will be removed from the unit.

By the way, the will offer you a command for the BI aggregation to acknowledge its problems, but this only means that all hosts and services detected in the aggregation will be acknowledged (only those which currently have problems).

Using the freeze function is very simple: Enable the New aggregations are frozen option in Aggregation Properties.

This will freeze the aggregation when it is saved — and will do so whenever the check mark is newly set; even if the aggregation has previously existed (despite the reference to New aggregations …​). To unfreeze the frozen status, remove the check mark accordingly.

In the monitoring, you will now see a new snowflake icon next to the aggregation. This will take you to the view showing the differences: On the left the frozen tree, on the right the current tree with the changes highlighted (here the removal of the service backup3):

If you again want to freeze the current state, you can do this via Commands > Freeze aggregations. But be careful: There is always only one frozen current state and no history including older states.

Can you actually notify of state changes in BI aggregations? Well — that’s not directly-possible at first, since BI exists exclusively in the GUI and has no relation to the actual monitoring. But you can turn BI aggregations into normal services, and these can in turn of course trigger notifications. There are two possibilities:

We will start with the data source program method, because this is always good if you wish to generate more than a handful of aggregations as services. You will find the appropriate rule set under Setup > Agents > Other integrations > BI Aggregations:

Here you can even specify different options for which hosts the services should be added. You do not necessarily have to stick to the host which is running the data source program (Assign to the querying host). It is also possible to assign to the hosts which are affected by the aggregation (Assign to the affected hosts). That however only makes sense if it concerns only a single Host. Regular expressions and substitutions can make you even more flexible with assignments. The whole thing is then performed via the piggyback mechanism.

Important: If the host to which you assign this rule should continue to be monitored through the normal agent, ensure in its settings that Agent and data source programs are run:

Notification with an active check is more or less the more direct way, and it requires no artificial 'helper host' when executing the data source program, since it has to query each unit individually, but with larger numbers of aggregations it is significantly less efficient and also more complicated to set up.

Putting it all simply: There is an active check which can retrieve the state of BI aggregations using HTTP from the REST API of Checkmk. You can easily set this up with the Setup > Services > Other services > Check State of BI Aggregation rule set:

Note the following:

In the example Automatically track downtimes of aggregation is activated. Strictly speaking, this means the scheduled downtimes — thus the planned maintenance times. This will make the new active service automatically get a maintenance time, even if the BI aggregation also does this!

The new service then shows — with a delay of up to one Check Interval of course — the state of the unit. The example shows the BI-Check on the host srv-mys-1:

As usual you can assign this service to contacts and use it as a basis for a notifications.