1. Introduction
Checkmk supports the concept of labels, of which you can assign any number to a host. Labels and host tags behave quite similarly:
Labels are 'attached' to hosts in the same way as tags.
Labels can be used as conditions for rules in the same way as tags.
Labels are created similarly to tags on the
key:value
principle.
So why is there a new concept here? Well, the IT world is changing and becoming much more dynamic. Cloud and container systems such as Amazon Web Services (AWS), Microsoft Azure and Kubernetes autonomously create and delete what they refer to as objects — which correspond to hosts in Checkmk. In these technologies labels and tags play a big role because they make the connections between the monitored objects and their functions. The host names, on the other hand, are increasingly random and meaningless.
Checkmk can create such dynamic hosts automatically with the dynamic host configuration, and can also receive information on any labels/tags that are already present. These labels/tags can then be used for rule conditions, searches, evaluations, dashboards, and other tasks.
Of course the question arises — why we do not simply map such dynamic labels to the existing concept of host tags. That is actually a very obvious conclusion at first look. However host tags have a very important property which would make that very difficult and complicated: Checkmk rigidly defines which tags and tag groups are present. Everything is well-defined. Every host has exactly one tag from every group. Everyone can rely on it. Typing errors in the spelling of tags cannot occur — not even with hosts which do not stick to the scheme — because its compliance is strictly controlled by Checkmk. This is very important and useful in very heterogeneous environments with many thousands of manually-managed hosts.
By contrast, dynamic labels from Kubernetes and Co are effectively 'free-form', and even if they do follow a scheme, this is unknown to Checkmk. In addition you might be monitoring several different platforms, which in turn use labels in very different ways.
That is why a Checkmk labels concept has been introduced which suits this growing dynamic. Of course you can also use the labels even if you do not use connections to Cloud environments.
To answer the question "When to use labels and when to use host tags?", there is more information in the article on structuring hosts.
Here are the features of labels:
Labels do not have to be predefined anywhere. There is no fixed scheme for labels. Everything is free-form. Everything is allowed.
Each host can have as many labels as you like. These can be maintained manually, defined via rules, or created automatically.
Labels are structured according to the
key:value
principle. Each host may only have one value per key. So a host that has the labelfoo:bar
cannot at the same time havefoo:bar2
.Unlike the host tags, both the key and the value — except for the colon (:) — may contain any characters.
There is no distinction between ID and title (or displayed name): the key of the label is both at the same time.
Labels have the following tasks:
They lay a foundation for conditions in configuration rules, for example: 'All hosts with the label
os:windows
should be monitored in the same way.'It is very easy to store additional information or comments about a host (for example,
location:RZ 74/123/xyz
) and to display this in views for example.
2. Creating labels
2.1. Explicit labels
Labels can be assigned to a host in a number of ways.
The first of these is simple: On the host properties page, which is displayed when you create or edit a host in the Setup, you can give a host as many labels as you like:
Activate Labels with the checkbox, then click in the Add some label field, enter the label definition following the scheme key:value
and finish with Enter.
You can edit an existing label by clicking it in its text, or delete it with the little 'x'.
Both the key and the value of a label may include any character — except the colon ( |
Of course labels can also be inherited from a folder.
Like other attributes, labels can be in subfolders or at the host, then as needed be overwritten again — individually in fact.
If for example, the label location:munich
is set in the folder, this will be inherited by all hosts in this folder which do not themselves already have the label location
defined.
Other labels a host may have will remain unaffected.
Explicitly-defined labels for hosts or folders appear violet in the list of hosts:
2.2. Creating labels using rules
As usual in Checkmk, attributes can also be assigned to hosts and services by rules. This will make you independent of the folder structure, and it also applies to the labels. There is a Host labels rule set for this function, which you can quickly find via a search in the Setup menu.
The following rule adds the hw:real
label to all hosts in the Bavaria
folder which have the Real Hardware
host tag:
You may have noticed that labels cannot be used in the conditions for this rule. This is not a mistake, but intentional, and it avoids recursive dependencies and other anomalies.
Labels added via rules appear red at the host shown, but do not appear in the host list in the Setup, instead they only appear with the status view of the host.
2.3. Automatically-generated labels
The third way labels can be created is fully-automatically. Various data sources such as the special agents for monitoring cloud and container systems automatically generate labels. In particular, the special agents for AWS, Azure and Kubernetes should be mentioned here. Sometimes these things are also referred to as tags on the respective platforms and are created in Checkmk as host or service labels. The respective rule sets provide sufficient information about this.
The nice thing is that you do not have to configure anything at all. As soon as these data sources are active, the corresponding labels will be generated.
In the Automatically-generated host labels section you will find an overview of the labels that Checkmk generates automatically.
2.4. Specifying labels via an agent plug-in
A simple way to manipulate labels directly is to add an agent plug-in, which, analogous to local checks, creates a section called labels
.
In this way you can assign labels in more detail than by evaluating the HW/SW inventory alone — for example, according to nuances of the installed hardware (such as CPU features), or relevant to processes that are actually running (instead of simply the installed software).
The label output is to be formatted as a Python dictionary, as in the following example:
<<<labels:sep(0)>>>
{"cpu/vendor": "zilog"}
Avoid conflicts with the labels automatically assigned by Checkmk itself and other plug-ins, as no particular order of evaluation can be guaranteed.
2.5. Including labels found in the discovery check
If you have enabled the discovery check — which is the default for new installations — it will warn you when new host labels are found that have not yet been added to the host properties in the Setup. This will look like this, for example:
You have two options for responding to this warning. The first is to add the new labels by calling the service configuration for the host in the Setup and updating the configuration of the labels with the Hosts > Update host labels menu item. The discovery check will then be OK again the next time it runs (up to a two-hour delay), even if you have not yet activated the changes. If you do not want to wait that long, you can also manually update the service immediately by selecting Reschedule check in the service’s action menu.
If this affects many hosts at once, you will certainly not want to visit the service configuration for each host. The best thing to do here is to run the bulk action for Run bulk service discovery and select Only discover new host labels as the mode — or alternatively Add unmonitored services and new host labels if you also want to take the opportunity to add new services.
The second way to turn the discovery check green is to reconfigure it so that it no longer warns you about new labels. To do this, go to the Periodic service discovery rule set, and edit the existing rule — there you will find the Severity of new host labels option:
This is set to Warning by default. Choose OK - do not alert, just display and the check will go silent.
Labels found via discovery check are displayed in yellow-ocher.
2.6. Sequence of label assignment priorities
Theoretically, the same label may be defined with different values in multiple sources simultaneously. That’s why there is the following order of priority:
First of all, explicit labels, i.e., those that you define for the host or folder directly in the Setup.
In the second place are labels that are created by rules.
In the last place are automatically-generated labels.
These precedence rules give you the ultimate control over the labels.
3. Labels as conditions in rules
3.1. Conditions in rules
An important function of labels is the same as for host features: Namely, their use as a condition in rules. This is especially useful for automatically-generated labels, because you can adapt your monitoring fully-automatically based on information from AWS, Azure, Kubernetes and co.
Add conditions with Add to condition for the host or service labels.
Now select either is or not to formulate a positive or negative condition and then enter the label in the usual form key:value
.
Pay attention to the exact spelling here, including capitalization.
As labels can be defined without specifications, Checkmk cannot recognize typos.
Nevertheless: When you type in a label, Checkmk suggests existing labels if they match your previous input.
No distinction is made between host and service labels in the suggestions: all matching labels are offered.
Pay attention to the correct spelling, as misspellings, incorrect capitalization, etc. will result in the rule no longer working.
However, the definition of the condition goes one step further:
To further refine the condition, you also have the Boolean operators Not
, And
and Or
at your disposal.
Here, Not
is the abbreviation for And Not
.
Not
means that condition A must be fulfilled and condition B must not be fulfilled at the same time.And
means that both condition A and condition B must be fulfilled at the same time.Or
means that either condition A or condition B must be fulfilled, but both conditions may also be fulfilled.
The operators are processed in exactly this priority — |
For example, to find hosts with the label cmk/site:today
but without the label cmk/piggyback_source_today:yes
, the condition could look like this:
You can refine this condition with is or not with any other specifications. Or you can add a new group of conditions with Add to condition, which makes the now more complex condition easier to read, but does not change the evaluation of the Boolean algebra:
If you have defined neither Host tags nor Host labels, the rule in question is always applied to all hosts or services. If you have created several rules, subsequent rules may no longer be evaluated, see Types of rule analysis. |
You can use both labels and host tags in a rule. These are automatically AND-linked. The rule therefore only applies if both conditions are fulfilled at the same time.
3.2. Conditions in notification rules
You can use labels as conditions in notification rules, too. The usage is the same as in other conditions, so you do not need to re-educate yourself to use them. Select Match host labels and simply enter which labels a host or service must have, thus triggering a notification through this rule. Again, multiple labels are connected by the AND operation.
4. Labels in views
So far we have only talked about the labels in the configuration environment (or short: the Setup). The labels are also visible in the monitoring environment — in the status view of a host, for instance:
Here you can see the labels in different colors depending on how they were created: Explicit labels in violet, labels created by rules in red and labels created by a discovery check in yellow-ocher.
The colored highlighting of the labels not only stands out visually in the view, they are also practical because they can be clicked on, which leads you to a search for all hosts with this label:
Here you can search for hosts with this label (using the default is) or without this label (using the option is not).
You can also use the Boolean operators Not
, And
and Or
in the label search, as described under Conditions in rules.
For example, to find Linux hosts that are located in Munich and which are not web servers, the filter could look like this:
You can further refine this filter, e.g. to additionally find Windows hosts that are both 'headless' (with or), and French (with and). You can enter the three new lines for this extension of the filter directly under the existing ones — or you can create a new group with Add to query, which makes the now more complex filter easier to read, but does not change the evaluation of the Boolean algebra:
If you are interested in which bracket replacement Checkmk generates from the label filters entered, you can have the associated Livestatus query displayed. To do this, activate Setup > General > Global settings > Debug Livestatus queries. |
In the filter bar you can of course also combine the search for labels with the other available search parameters.
5. Service labels
Services can also have labels. These are similar to the host labels, however with a few small differences:
You cannot define service labels explicitly. These can only be created by rules (Service labels), or automatically.
You can also formulate conditions with service labels. In a rule set, the service labels are only offered for input if the rule can match a service.
6. Agent labels
Checkmk Cloud includes the option to create hosts automatically. For this, the whole chain of agent registration, host creation, service discovery and change activation is automated. The creation of the host takes place following the registration.
This automation creates some challenges for the structuring of newly created hosts.
Until now, the operating system (stored in the host label cmk/os_family
) could only be determined from the agent output.
However, to get this, the host must already have been created.
For this reason, the concept of volatile agent labels was introduced. These labels are submitted during a registration and are thus available before the first agent is created. During registration, you can use the agent labels to determine whether a host should actually be created in the monitoring, and — if so — to influence the host’s folder, as well as other host attributes. Once registration has been completed, agent labels can no longer be accessed.
Two predefined agent labels are always transferred during a registration:
cmk/os-family
contains the operating system family (currentlyWindows
orLinux
).cmk/hostname-simple
contains the computer name in an abbreviated form (i.e. without the domain portion)
You can freely assign additional, custom agent labels, for example:
organizational/department:documentation
.
Automatically-registered hosts are assigned the cmk/agent_auto_registered:yes
host label.
A direct assignment of host labels as a result of specific agent labels is not supported.
However, you have the alternative of registering hosts in a (temporary) folder and then assigning host labels for all hosts in that folder.
7. Further information
7.1. Automatically-generated host labels
Key | Values |
---|---|
|
|
|
|
|
Name of the AWS account the host belongs to |
|
Tags from AWS objects |
|
Resource group the Azure object belongs to |
|
Tags from Azure objects |
|
|
|
|
|
SNMP transmitted device name, e.g. |
|
Docker image, e.g. |
|
Name of the Docker image, e.g. |
|
Version of the Docker image, e.g. |
|
|
|
These labels are output for any Kubernetes label that is a valid Kubernetes annotation (configurable via the Kubernetes rule). |
|
|
|
name of the Kubernetes cluster |
|
name of the Kubernetes cluster host |
|
Kubernetes Cronjob |
|
Name of the DaemonSet |
|
Name of the deployment |
|
Name of the associated Kubernetes namespace |
|
Name of the associated Kubernetes node. Checkmk hosts of type Pod or Node get this label. |
|
Kubernetes object type, e.g. |
|
Name of the StatefulSet |
|
|
|
Meraki device type, e.g. |
|
Network ID the Meraki device belongs to |
|
Organization ID the Meraki device belongs to |
|
Organization name the Meraki device belongs to |
|
|
|
Operating system, reported by the agent as |
|
Operating system type, reported by the agent as |
|
Operating system name, reported by the agent as |
|
Operating system platform, reported by the agent as |
|
Operating system version, reported by the agent as |
|
|
|
|