Checkmk
to checkmk.com

1. Introduction

Checkmk supports the concept of labels, of which you can assign any number to a host. Labels and host tags behave quite similarly:

  • Labels are ‘attached’ to hosts in the same way as tags.

  • Labels can be used as conditions for rules in the same way as tags.

  • Labels are constructed similarly to tags on the key:value principle.

So why is there a new concept here? Well, the IT world is changing and becoming much more dynamic. Cloud and container systems such as Amazon Web Services (AWS), Microsoft Azure and Kubernetes autonomously create and delete objects – which correspond to hosts in Checkmk. In these technologies labels and tags play a big role because they make the connections between the monitored objects and their functions. The host names, on the other hand, are increasingly random and meaningless.

Checkmk can create such dynamic hosts automatically with the dynamic host configuration, and can also receive information on any labels/tags that are already present. These labels/tags can then be used for rule conditions, searches, evaluations, dashboards, and other tasks.

Of course the question arises — why we do not simply map such dynamic labels to the existing concept of host tags. In fact that is also a very obvious conclusion at first look. However host tags have a very important property which would make that very difficult and complicated: Checkmk rigidly defines which tags and tag groups are present. Everything is well-defined. Every host has exactly one tag from every group. Everyone can rely on it. Typing errors in the spelling of tags cannot occur — not even with hosts which do not stick to the scheme — because its compliance is strictly controlled by Checkmk. This is very important and useful in very heterogeneous environments with many thousands of manually-managed hosts.

By contrast, dynamic labels from Kubernetes and Co are effectively ‘freeform’, and even if they do follow a scheme, this is unknown to Checkmk. In addition you might be monitoring several different platforms, which in turn use labels in very different ways.

That is why a Checkmk labels concept has been introduced which suits this growing dynamic. Of course you can also use the labels even if you don’t use connections to Cloud environments.

To answer the question "When to use labels and when to use host tags?", there is more information in the article on structuring hosts.

Here are the features of labels:

  • Labels do not have to be predefined anywhere. There is no fixed scheme for labels. Everything is free form. Everything is allowed.

  • Each host can have as many labels as you like. These can be maintained manually, defined via rules, or created automatically.

  • Labels are structured according to the key:value principle. Each host may only have one value per key. So a host that has the label foo:bar cannot at the same time have foo:bar2.

  • Unlike the host tags, both the key and the value — except for the colon (:) — may contain any characters.

  • There is no distinction between ID and title (or displayed name): the key of the label is both at the same time.

Labels have the following tasks:

  • They form a basis for conditions in configuration rules, for example: 'All hosts with the label os:windows should be monitored in the same way.'

  • It is very easy to store additional information or comments about a host (for example, location:RZ 74/123/xyz) and to display this in views for example.

2. Creating labels

2.1. Explicit labels

Labels can be assigned to a host in a number of ways.

The first of these is simple: On the host properties page, which is displayed when you create or edit a host in the Setup, you can give it as many labels as you like:

Dialog with properties of a host for defining labels.

Activate Labels with the checkbox, then click in the Add some label field, enter the label definition in the form key:value and finish with Enter.

You can edit an existing label by clicking it in its text, or delete it with the little cross.

Note: Both the key and the value of a label may include any character — except the colon (:)! However you should think carefully about the use of characters and upper/lower case, since if you later define conditions via labels, then the spelling of both the key and the value must be strictly observed.

Of course labels can also be inherited from a folder. Like other attributes, labels can be in subfolders or at the host, then as needed be overwritten again — individually in fact. If for example, the label location:munich is set in the folder, this will be inherited by all hosts in this folder which do not themselves already have the label location defined. Other labels a host may have will remain unaffected.

Explicitly-defined labels for hosts or folders appear violet in the list of hosts:

List entry of a host with the assigned explicit labels.

2.2. Creating labels using rules

As usual in Checkmk, attributes can also be assigned to hosts and services by rules. This will make you independent of the folder structure, and it also applies to the labels. There is a Host labels rule set for this function, which you can quickly find via a search in the Setup menu.

The following rule adds the hw:real label to all hosts in the Bavaria folder which have the Real Hardware host tag:

Rule for specifying labels for hosts.

You may have noticed that labels cannot be used in the conditions for this rule. This is not a mistake, but intentional, and it avoids recursive dependencies and other anomalies.

Labels added via rules appear red at the host shown, but do not appear in the host list in the Setup, instead they only appear with the status view of the host.

2.3. Automatically generated labels

The third way labels can be created is fully automatically. Various data sources such as the special agents for monitoring cloud and container systems, as well as the agent plug-in of the hardware/software inventory automatically generate labels.

The nice thing is that you do not have to configure anything at all. As soon as these data sources are active, the corresponding labels will be generated.

In the Automatically generated host labels section you will find an overview of the labels that Checkmk generates automatically.

2.4. Specifying labels via an agent plug-in

A simple way to manipulate labels directly is to add an agent plug-in, which, analogous to local checks, creates a section called labels. In this way you can assign labels in more detail than by evaluating the HW/SW inventory alone — for example, according to nuances of the installed hardware (such as CPU features) or actually running processes (instead of just installed software).

The label output is to be formatted as a Python dictionary, as in the following example:

<<<labels:sep(0)>>>
{"cpu/vendor": "zilog"}

Avoid conflicts with the labels automatically assigned by Checkmk itself and other plug-ins, as no particular order of evaluation can be guaranteed.

2.5. Including labels found in the Discovery Check

If you have enabled the discovery check — which is the default for new installations — it will warn you when new host labels are found that have not yet been added to the host properties in the Setup. This will look like this, for example:

Service list with the service 'Check_MK Discovery'.

You have two options for responding to this warning. The first is to add the new labels by calling the service configuration for the host in the Setup and updating the configuration of the labels with the Hosts > Update host labels menu item. The discovery check will then be OK again the next time it runs (up to a two-hour delay), even if you have not yet activated the changes. If you don’t want to wait that long, you can also manually update the service immediately by selecting Reschedule check in the action menu of the service.

If this affects many hosts at once, you will certainly not want to visit the service configuration for each host. The best thing to do here is to run the bulk action for Discover services and select Only discover new host labels as the mode — or alternatively Add unmonitored services and new host labels if you also want to take the opportunity to add new services.

The second way to turn the discovery check green is to reconfigure it so that it no longer warns you about new labels. To do this, go to the Periodic service discovery rule set, and edit the existing rule — there you will find the Severity of new host labels option:

Rule for periodic service discovery.

This is set to Warning by default. Choose OK - do not alert, just display and the check will go silent.

Labels found via discovery check are displayed in orange.

2.6. Sequence of label assignment priorities

Theoretically, the same label may be defined with different values in multiple sources simultaneously. That’s why there is the following order of priority:

  1. First of all, explicit labels, i.e., those that you define for the host or folder directly in the Setup.

  2. In the second place are labels that are created by rules.

  3. In the last place are automatically-generated labels.

These precedence rules give you the ultimate control over the labels.

3. Labels as conditions in rules

3.1. Conditions in rules

An important function of labels is the same as with tags, namely their ‘Use’ condition in rules. This is especially true for automatically-generated labels, because they perform their monitoring fully-automatically according to information from AWS, Azure, Kubernetes, etc.

The following example shows a rule condition that only applies when the host has the label state:bavaria, but not the label environment:test:

Dialog for defining conditions with labels.

You can use both labels and host tags in a rule. These will be automatically AND-linked. The rule only applies if both conditions are met simultaneously.

Note that the exact spelling of the labels is important. Since labels can be defined without any defaults, Checkmk cannot recognize typos. Nevertheless: When typing a label, Checkmk suggests already existing labels if they match your previous input. The suggestions do not distinguish between host and service labels, so all matching labels are offered.

Tip: If this causes isolated problems it may be more effective if you work with host tags. See the structuring hosts article for more information about the different structuring elements.

3.2. Conditions in notification rules

You can use labels as conditions in notification rules, too. The usage is the same as in other conditions, so you don’t need re-educate yourself to use them. Select Match host labels and simply enter which labels a host or service must have, thus triggering a notification through this rule. Again, multiple labels are connected by the AND operation.

4. Labels in views

So far we have only talked about the labels in the Setup (or the configuration environment). The labels are also visible in the monitoring environment — in the status view of a host, for instance:

Dialog with the host status and the assigned labels.

Here you can see the labels in different colors depending on how they were created: Explicit labels in violet, labels created by rules in red and labels created by a discovery check in yellow-ocher.

The colored highlighting of the labels not only stands out visually in the view, they are also practical because they can be clicked on, which leads you to a search for all hosts with this label:

Filter bar with filter to search for a label.

Here you can search for hosts with this label (using the default is) or without this label (using the option is not).

However, the label search goes one step further: To further refine the result list from your hosts, the Boolean operators Not, And and Or are also available. Here Not is the abbreviation for And Not.

Important: The operators are processed in exactly this priority — Not, And, Or — so not necessarily in the order they are in the filter’s list. This conforms to the Boolean algebra standard.
For example, A And B Not C Or D would correspond to (A And (B Not C)) Or D.

To find Linux hosts that are located in Munich and which are not web servers, the filter could look like this, for example:

Filter options with 3 label filters linked by logical operators.

You can further refine this filter, e.g. to additionally find Windows hosts that are both 'headless' (with Or), and French (with And). You can enter the three new lines for this extension of the filter directly under the existing ones — or you can create a new group with Add to query, which makes the now more complex filter easier to read, but does not change the evaluation of the Boolean algebra:

Filter options with 6 label filters linked by logical operators.

Note: If you are interested in which bracket replacement Checkmk generates from the label filters entered, you can have the associated Livestatus query displayed. To do this, activate Setup > General > Global settings > Debug Livestatus queries.

Of course, you can also combine the search for labels with the other available search parameters in the filter bar.

5. Service labels

Services can also have labels. These are similar to the host labels, however with a few small differences:

  • You cannot define service labels explicitly. These can only be created by rules (Service labels), or automatically.

  • You can also formulate conditions with service labels. In a rule set, the service labels are only offered for input if the rule can match a service.

6. Agent labels

The CSE Checkmk Cloud Edition includes the option to create hosts automatically. For this, the whole chain of agent registration, host creation, service discovery and change activation is automated. The creation of the host takes place following the registration.

This automation creates some challenges for the structuring of newly created hosts. Until now, the operating system (stored in the host label cmk/os_family) could only be determined from the agent output. However, to get this, the host must already have been created.

For this reason, the concept of volatile agent labels was introduced. These labels are submitted during a registration and are thus available before the first agent is created. During registration, you can use the agent labels to determine whether a host should actually be created in the monitoring, and — if so — to influence the host’s folder, as well as other host attributes. Once registration has been completed, agent labels can no longer be accessed.

Two predefined agent labels are always transferred during a registration:

  • cmk/os-family contains the operating system family (currently Windows or Linux).

  • cmk/hostname-simple contains the computer name in an abbreviated form (i.e. without the domain portion)

You can freely assign additional, custom agent labels, for example: organizational/department:documentation.

Automatically registered hosts are assigned the host label cmk/agent_auto_registered:yes. A direct assignment of host labels as a result of specific agent labels is not supported. However, you have the alternative of registering hosts in a (temporary) folder and then assigning host labels for all hosts in that folder.

7. Further information

7.1. Automatically generated host labels

Key Values

cmk/agent_auto_registered

yes, if a host was created via auto-registration

cmk/check_mk_server

yes for all Checkmk servers

cmk/device_type

SNMP transmitted device name, e.g. appliance, fcswitch, firewall, printer, router, sensor, switch, ups, wlc.

cmk/docker_image

Docker image, e.g. docker.io/library/nginx:latest

cmk/docker_image_name

Name of the Docker image, e.g. nginx.

cmk/docker_image_version

Version of the Docker image, e.g. latest.

cmk/docker_object

container, if the host is a Docker container; node, if the host is a Docker node

cmk/kubernetes/annotation/{key}:{value}

These labels are output for any Kubernetes label that is a valid Kubernetes annotation (configurable via the Kubernetes rule).

cmk/kubernetes

yes if the host is a Kubernetes object.

cmk/kubernetes/cluster

Kubernetes cluster name

cmk/kubernetes/daemonset

Name of the DaemonSet

cmk/kubernetes/deployment

Name of the deployment

cmk/kubernetes/namespace

Name of the associated Kubernetes namespace

cmk/kubernetes/node

Name of the associated Kubernetes node. Checkmk hosts of type Pod or Node get this label.

cmk/kubernetes/object

Kubernetes object type, e.g. endpoint if the host is a Kubernetes endpoint object.

cmk/kubernetes/statefulset

Name of the StatefulSet

cmk/os_family

Operating system, reported by the agent as AgentOS (e.g. windows or linux )

cmk/vsphere_object

vm if the host is a virtual machine; server if the host is an ESXi host system.

cmk/vsphere_vcenter

yes if the host is a VMware vCenter.

7.2. Check plug-ins in which labels are registered

You can alternatively populate the Kubernetes-related cmk/kubernetes and cmk/kubernetes/object keys using the following plug-ins when using the Checkmk special agent for Prometheus to fetch the so-called 'kube-state-metrics':

  • k8s_daemon_pods

  • k8s_ingress_infos

  • k8s_job_info

  • k8s_nodes

  • k8s_pod_container

  • k8s_replicas

  • k8s_service_port

  • k8s_stateful_set_replicas

On this page