Checkmk
to checkmk.com

1. Introduction

Checkmk supports the concept of labels, of which you can assign any number to a host. Labels and host tags behave quite similarly:

  • Labels are ‘attached’ to hosts in the same way as tags.

  • Labels can be used as conditions for rules in the same way as tags.

  • Labels are constructed similarly to tags on the key:value principle.

So why is there a new concept here? Well, the IT world is changing and becoming much more dynamic. Cloud and container systems such as Amazon Web Services (AWS), Microsoft Azure and Kubernetes autonomously create and delete objects – which correspond to hosts in Checkmk. In these technologies labels and tags play a big role because they make the connections between the monitored objects and their functions. The host names, on the other hand, are increasingly random and meaningless.

Checkmk can create such dynamic hosts automatically with the dynamic host configuration, and can also receive information on any labels/tags that are already present. These labels/tags can then be used for rule conditions, searches, evaluations, dashboards, and other tasks.

Of course the question arises — why we do not simply map such dynamic labels to the existing concept of host tags. In fact that is also a very obvious conclusion at first look. However host tags have a very important property which would make that very difficult and complicated: Checkmk rigidly defines which tags and tag groups are present. Everything is well-defined. Every host has exactly one tag from every group. Everyone can rely on it. Typing errors in the spelling of tags cannot occur — not even with hosts which do not stick to the scheme — because its compliance is strictly controlled by Checkmk. This is very important and useful in very heterogeneous environments with many thousands of manually-managed hosts.

By contrast, dynamic labels from Kubernetes and Co are effectively 'freeform', and even if they do follow a scheme, this is unknown to Checkmk. In addition you might be monitoring several different platforms, which in turn use labels in very different ways.

That is why a Checkmk labels concept has been introduced which suits this growing dynamic. Of course you can also use the labels even if you don’t use connections to Cloud environments.

To answer the question "When to use labels and when to use host tags?", there is more information in the article on structuring hosts.

Here are the features of labels:

  • Labels do not have to be predefined anywhere. There is no fixed scheme for labels. Everything is free form. Everything is allowed.

  • Each host can have as many labels as you like. These can be maintained manually, defined via rules, or created automatically.

  • Labels are structured according to the key:value principle. Each host may only have one value per key. So a host that has the label foo:bar cannot at the same time have foo:bar2.

  • Unlike the host tags, both the key and the value — except for the colon (:) — may contain any characters.

  • There is no distinction between ID and title (or displayed name): the key of the label is both at the same time.

Labels have the following tasks:

  • They form a basis for conditions in configuration rules, for example: 'All hosts with the label os:windows should be monitored in the same way.'

  • It is very easy to store additional information or comments about a host (for example, location:RZ 74/123/xyz) and to display this in views for example.

2. Creating labels

2.1. Explicit labels

Labels can be assigned to a host in a number of ways.

The first of these is simple: On the host properties page, which is displayed when you create or edit a host in the Setup, you can give it as many labels as you like:

Dialog with properties of a host for defining labels.

Activate Labels with the checkbox, then click in the Add some label field, enter the label definition in the form key:value and finish with Enter.

You can edit an existing label by clicking it in its text, or delete it with the little cross.

Note: Both the key and the value of a label may include any character — except the colon (:)! However you should think carefully about the use of characters and upper/lower case, since if you later define conditions via labels, then the spelling of both the key and the value must be strictly observed.

Of course labels can also be inherited from a folder. Like other attributes, labels can be in subfolders or at the host, then as needed be overwritten again — individually in fact. If for example, the label location:munich is set in the folder, this will be inherited by all hosts in this folder which do not themselves already have the label location defined. Other labels a host may have will remain unaffected.

Explicitly-defined labels for hosts or folders appear violet in the list of hosts:

List entry of a host with the assigned explicit labels.

2.2. Creating labels using rules

As usual in Checkmk, attributes can also be assigned to hosts and services by rules. This will make you independent of the folder structure, and it also applies to the labels. There is a Host labels rule set for this function, which you can quickly find via a search in the Setup menu.

The following rule adds the hw:real label to all hosts in the Bavaria folder which have the Real Hardware host tag:

Rule for specifying labels for hosts.

You may have noticed that labels cannot be used in the conditions for this rule. This is not a mistake, but intentional, and it avoids recursive dependencies and other anomalies.

Labels added via rules appear red at the host shown, but do not appear in the host list in the Setup, instead they only appear with the status view of the host.

2.3. Automatically generated labels

The third way labels can be created is fully automatically. Various data sources such as the special agents for monitoring cloud and container systems automatically generate labels. In particular, the special agents for AWS, Azure and Kubernetes should be mentioned here. Sometimes these things are also referred to as tags on the respective platforms and are created in Checkmk as host or service labels. The respective rule sets provide sufficient information about this.

The nice thing is that you do not have to configure anything at all. As soon as these data sources are active, the corresponding labels will be generated.

In the Automatically generated host labels section you will find an overview of the labels that Checkmk generates automatically.

2.4. Specifying labels via an agent plug-in

A simple way to manipulate labels directly is to add an agent plug-in, which, analogous to local checks, creates a section called labels. In this way you can assign labels in more detail than by evaluating the HW/SW inventory alone — for example, according to nuances of the installed hardware (such as CPU features) or actually running processes (instead of just installed software).

The label output is to be formatted as a Python dictionary, as in the following example:

<<<labels>>>
{"cpu/vendor": "zilog"}

Avoid conflicts with the labels automatically assigned by Checkmk itself and other plug-ins, as no particular order of evaluation can be guaranteed.

2.5. Including labels found in the Discovery Check

If you have enabled the discovery check — which is the default for new installations — it will warn you when new host labels are found that have not yet been added to the host properties in the Setup. This will look like this, for example:

Service list with the service 'Check_MK Discovery'.

You have two options for responding to this warning. The first is to add the new labels by calling the service configuration for the host in the Setup and updating the configuration of the labels with the Hosts > Update host labels menu item. The discovery check will then be OK again the next time it runs (up to a two-hour delay), even if you have not yet activated the changes. If you don’t want to wait that long, you can also manually update the service immediately by selecting Reschedule check in the action menu of the service.

If this affects many hosts at once, you will certainly not want to visit the service configuration for each host. The best thing to do here is to run the bulk action for Run bulk service discovery and select Only discover new host labels as the mode — or alternatively Add unmonitored services and new host labels if you also want to take the opportunity to add new services.

The second way to turn the discovery check green is to reconfigure it so that it no longer warns you about new labels. To do this, go to the Periodic service discovery rule set, and edit the existing rule — there you will find the Severity of new host labels option:

Rule for periodic service discovery.

This is set to Warning by default. Choose OK - do not alert, just display and the check will go silent.

Labels found via discovery check are displayed in yellow-ocher.

2.6. Sequence of label assignment priorities

Theoretically, the same label may be defined with different values in multiple sources simultaneously. That’s why there is the following order of priority:

  1. First of all, explicit labels, i.e., those that you define for the host or folder directly in the Setup.

  2. In the second place are labels that are created by rules.

  3. In the last place are automatically-generated labels.

These precedence rules give you the ultimate control over the labels.

3. Labels as conditions in rules

3.1. Conditions in rules

An important function of Labels is the same as for host features: Namely, their use as a condition in rules. This is especially useful for automatically generated labels, because you can adapt your monitoring fully automatically based on information from AWS, Azure, Kubernetes and co.

Add conditions with Add to condition for the host or service labels. Now select either is or not to formulate a positive or negative condition and then enter the label in the usual form key:value. Pay attention to the exact spelling here, including capitalization. As labels can be defined without specifications, Checkmk cannot recognize typos. Nevertheless: When you type in a label, Checkmk suggests existing labels if they match your previous input. No distinction is made between host and service labels in the suggestions: all matching labels are offered. Otherwise the condition will not work correctly.

Tip: If this causes difficulties in individual cases, it may be better to work with host characteristics. In the article on Structuring the hosts you will find more information on the various structuring elements.

However, the definition of the condition goes one step further: To further refine the condition, you also have the Boolean operators Not, And and Or at your disposal. Here, Not is the abbreviation for And Not.

  • Not means that condition A must be fulfilled and condition B must not be fulfilled at the same time.

  • And means that both condition A and condition B must be fulfilled at the same time.

  • Or means that either condition A or condition B must be fulfilled, but both conditions may also be fulfilled.

Important: The operators are processed in exactly this priority — Not, And, Or — i.e. not necessarily in the order in which they appear in the list. This corresponds to the standard of Boolean algebra. For example, A And B Not C Or D would correspond to the parentheses (A And (B Not C)) Or D.

For example, to find hosts with the label cmk/site:today but without the label cmk/piggyback_source_today:yes, the condition could look like this:

Condition for host labels.

You can refine this condition with is or not with any other specifications. Or you can add a new group of conditions with Add to condition, which makes the now more complex condition easier to read, but does not change the evaluation of the Boolean algebra:

multiple conditions for host labels.

Note: If you have defined neither Host tags nor Host labels, the rule in question is always applied to all hosts or services. If you have created several rules, subsequent rules may no longer be evaluated, see Types of rule analysis.

You can use both labels and host characteristics in a rule. These are automatically AND-linked. The rule therefore only applies if both conditions are fulfilled at the same time.

3.2. Conditions in notification rules

You can use labels as conditions in notification rules, too. The usage is the same as in other conditions, so you don’t need re-educate yourself to use them. Select Match host labels and simply enter which labels a host or service must have, thus triggering a notification through this rule. Again, multiple labels are connected by the AND operation.

4. Labels in views

So far we have only talked about the labels in the Setup (or the configuration environment). The labels are also visible in the monitoring environment — in the status view of a host, for instance:

Dialog with the host status and the assigned labels.

Here you can see the labels in different colors depending on how they were created: Explicit labels in violet, labels created by rules in red and labels created by a discovery check in yellow-ocher.

The colored highlighting of the labels not only stands out visually in the view, they are also practical because they can be clicked on, which leads you to a search for all hosts with this label:

Filter bar with filter to search for a label.

Here you can search for hosts with this label (using the default is) or without this label (using the option is not).

You can also use the Boolean operators Not, And and Or in the label search, as described under Conditions in rules. For example, to find Linux hosts that are located in Munich and are not web servers, the filter could look like this:

Filter options with 3 label filters linked by logical operators.

You can further refine this filter, e.g. to additionally find Windows hosts that are both 'headless' (with Or), and French (with And). You can enter the three new lines for this extension of the filter directly under the existing ones — or you can create a new group with Add to query, which makes the now more complex filter easier to read, but does not change the evaluation of the Boolean algebra:

Filter options with 6 label filters linked by logical operators.

Note: If you are interested in which bracket replacement Checkmk generates from the label filters entered, you can have the associated Livestatus query displayed. To do this, activate Setup > General > Global settings > Debug Livestatus queries.

Of course, you can also combine the search for labels with the other available search parameters in the filter bar.

5. Service labels

Services can also have labels. These are similar to the host labels, however with a few small differences:

  • You cannot define service labels explicitly. These can only be created by rules (Service labels), or automatically.

  • You can also formulate conditions with service labels. In a rule set, the service labels are only offered for input if the rule can match a service.

6. Agent labels

The CSE Checkmk Cloud Edition includes the option to create hosts automatically. For this, the whole chain of agent registration, host creation, service discovery and change activation is automated. The creation of the host takes place following the registration.

This automation creates some challenges for the structuring of newly created hosts. Until now, the operating system (stored in the host label cmk/os_family) could only be determined from the agent output. However, to get this, the host must already have been created.

For this reason, the concept of volatile agent labels was introduced. These labels are submitted during a registration and are thus available before the first agent is created. During registration, you can use the agent labels to determine whether a host should actually be created in the monitoring, and — if so — to influence the host’s folder, as well as other host attributes. Once registration has been completed, agent labels can no longer be accessed.

Two predefined agent labels are always transferred during a registration:

  • cmk/os-family contains the operating system family (currently Windows or Linux).

  • cmk/hostname-simple contains the computer name in an abbreviated form (i.e. without the domain portion)

You can freely assign additional, custom agent labels, for example: organizational/department:documentation.

Automatically registered hosts are assigned the host label cmk/agent_auto_registered:yes. A direct assignment of host labels as a result of specific agent labels is not supported. However, you have the alternative of registering hosts in a (temporary) folder and then assigning host labels for all hosts in that folder.

7. Further information

7.1. Automatically generated host labels

Key Values

cmk/agent_auto_registered

yes, if a host was created via auto-registration

cmk/aws/ec2

instance for all EC2 instances

cmk/aws/account

Name of the AWS account the host belongs to

cmk/aws/tag/{key}:{value}

Tags from AWS objects

cmk/azure/resource_group

Resource group the Azure object belongs to

cmk/azure/tag/{key}:{value}

Tags from Azure objects

cmk/azure/vm

instance for all Azure VMs

cmk/check_mk_server

yes for all Checkmk servers

cmk/device_type

SNMP transmitted device name, e.g. appliance, fcswitch, firewall, printer, router, sensor, switch, ups, wlc.

cmk/docker_image

Docker image, e.g. docker.io/library/nginx:latest

cmk/docker_image_name

Name of the Docker image, e.g. nginx.

cmk/docker_image_version

Version of the Docker image, e.g. latest.

cmk/docker_object

container, if the host is a Docker container; node, if the host is a Docker node

cmk/kubernetes/annotation/{key}:{value}

These labels are output for any Kubernetes label that is a valid Kubernetes annotation (configurable via the Kubernetes rule).

cmk/kubernetes

yes if the host is a Kubernetes object.

cmk/kubernetes/cluster

name of the Kubernetes cluster

cmk/kubernetes/cluster-host

name of the Kubernetes cluster host

cmk/kubernetes/cronjob

Kubernetes Cronjob

cmk/kubernetes/daemonset

Name of the DaemonSet

cmk/kubernetes/deployment

Name of the deployment

cmk/kubernetes/namespace

Name of the associated Kubernetes namespace

cmk/kubernetes/node

Name of the associated Kubernetes node. Checkmk hosts of type Pod or Node get this label.

cmk/kubernetes/object

Kubernetes object type, e.g. endpoint if the host is a Kubernetes endpoint object.

cmk/kubernetes/statefulset

Name of the StatefulSet

cmk/meraki

yes for all Meraki devices

cmk/meraki/device_type

Meraki device type, e.g. switch or wireless

cmk/meraki/net_id

network ID the Meraki device belongs to

cmk/meraki/org_id

organization ID the Meraki device belongs to

cmk/meraki/org_name

organization name the Meraki device belongs to

cmk/nutanix/object

control_plane for the host with the Nutanix special agent; node for a Nutanix host; vm for Nutanix VMs

cmk/os_family

Operating system, reported by the agent as AgentOS (e.g. windows or linux )

cmk/os_type

Operating system type, reported by the agent as OS type (e.g. windows, linux or unix)

cmk/os_name

Operating system name, reported by the agent as OSName (e.g. Microsoft Windows 10 Pro, Ubuntu or Oracle Solaris)

cmk/os_platform

Operating system platform, reported by the agent as OSPlatform (e.g. Ubuntu for Ubuntu derivatives such as Xubuntu), if stored in /etc/os-release; if this line is missing in the agent output, the label receives the value of cmk/os_family.

cmk/os_version

Operating system version, reported by the agent as OSVersion (e.g. 22.04 for Ubuntu or 10.0.19045 for Windows)

cmk/vsphere_object

vm if the host is a virtual machine; server if the host is an ESXi host system.

cmk/vsphere_vcenter

yes if the host is a VMware vCenter.

On this page