1. Foreword

There may still be changes to both this description and the new Kubernetes monitoring feature in Checkmk version 2.1.0 itself. Please see the Changes to this article on GitHub or our Werks regarding changes to the feature itself.

Since the Kubernetes integration with Checkmk is natively built into Kubernetes itself, we also use the README files directly in the GitHub repositories. In particular, the Instructions for installing the agent is the direct source for reading up on the currently recommended installation procedures.

1.1. Getting started with the Kubernetes monitoring

For an introduction to the new monitoring of Kubernetes, we recommend the two videos Kubernetes Monitoring with Checkmk and Detecting issues and configuring alerts for Kubernetes clusters.

1.2. Differences from the previous Kubernetes monitoring

Kubernetes monitoring in Checkmk has been rewritten from scratch. The volume of data that can be monitored has significantly increased. Since the technical basis of Kubernetes monitoring is fundamentally different in Checkmk 2.1.0, it is not possible to adopt or even rewrite previous monitoring data for your Kubernetes objects.

2. Introduction

Kubernetes has been the most widely used tool for container orchestration for quite some time. Checkmk helps you monitor your Kubernetes environments.

Starting with version 2.1.0, you can use Checkmk to monitor the following Kubernetes objects:

  • Clusters

  • Nodes

  • Deployments

  • Pods

  • DaemonSets

  • StatefulSets

For a complete listing of all of the check plug-ins available for Kubernetes monitoring, see our Catalog of Check Plug-ins.

3. Prerequisites in the cluster

To be able to monitor your Kubernetes cluster in Checkmk, first create the prerequisites in your cluster. First and foremost, tell the cluster which pods/containers to deploy and how to configure them.

3.1. Setting up the Helm repository

We currently recommend installing Kubernetes monitoring using the helm tool, as it is also suitable for less experienced users and standardizes the management of configurations. Helm is a kind of a package manager for Kubernetes. You can use it to include repositories as sources and easily add the Helm charts they contain like packages to your cluster. To do this, first of all make the repository known. In the following example, we use the name tribe29 to make it easier to access the repository later. However, you can of course use any other name of your choice:

3.2. Customizing the configuration

Using Helm, you create the necessary configuration files completely independently. In order to be able to determine certain parameters over all configurations, you specify a control file, the so-called values.yaml. As a starting point we recommend the template provided by us. Copy it and adapt it to your own environment.

Since we cannot know in advance how your Kubernetes cluster is set up, we have chosen the safest option for how the Checkmk collectors are started: By default, they do not expose any ports to be reached from the outside. To allow you to access the collectors later, adjust these settings accordingly.

For simplicity, let’s take our template as a starting point. We support two communication paths by default: the query via Ingress and the query via NodePort. Depending on which variant you support in your cluster, the configuration will vary.

Providing communication via Ingress

If you use Ingress to control access to your services, adapt the already prepared parts in values.yaml accordingly. For a better overview only the relevant part is shown in the following example. Set the value enabled to true. You adjust the remaining values according to your environment:

    enabled: true
    className: ""
    annotations: /
      - host: checkmk-cluster-collector.local
          - path: /
            pathType: Prefix
    tls: []
    #  - secretName: chart-example-tls
    #    hosts:
    #      - chart-example.local

Providing communication via NodePort

You can also provide access to the services directly through a port. This is necessary if you do not use Ingress. Also in the following example only the relevant section is shown. You set the value type to NodePort and remove the comment for the value nodePort:

    # if required specify "NodePort" here to expose the cluster-collector via the "nodePort" specified below
    type: NodePort
    port: 8080
    nodePort: 30035

3.3. Creating the configuration files

After customizing values.yaml or creating your own files, use the following command to create all of the necessary configuration files to set up your Kubernetes cluster for monitoring in Checkmk:

user@host:~$ helm upgrade --install --create-namespace -n checkmk-monitoring checkmk tribe29/checkmk -f values.yaml

Since the command is not self-explanatory, we will provide an explanation of each option below:

Command parameterFunction

helm upgrade --install

This parameter is the basic command to send the configuration to the Kubernetes cluster.


In Kubernetes you always specify to which namespace the configuration should be added. You need this option if the namespace does not already exist. Helm will create it in this case.

-n checkmk-monitoring

This option specifies the namespace to which the configuration should be added. checkmk-monitoring is just an example of what it could be called.


checkmk is an example for the name of the Helm chart. Ideally, you should leave this name as it is, because only then will you automatically benefit from the fact that Kubernetes objects get short names.


The first part of this option describes the repository you created with the previous command. The second part — after the slash — is the package where the necessary information is located to be able to create the configuration of your Kubernetes monitoring.

-f values.yaml

Finally, specify the configuration file that you created or customized earlier. It contains all of the customizations to be included in the configuration files created with helm.

After you run the command, your Kubernetes cluster will be ready to be monitored with Checkmk. The cluster will now take care of itself to ensure that the necessary pods and the containers within them are running and accessible.

3.4. Alternative: Setting up via manifest

It does not normally make sense for you to customize the manifests (configuration files) yourself. On the one hand, because you need detailed knowledge of the architecture of the Checkmk Kubernetes Collectors for this and, on the other hand, because manual customization is much more error-prone. For example, in helm you set up communication over TLS once only, instead of needing to add it to all relevant places in each actual individual manifest.

However, if you don`t use helm, or want to have control over all the setup details, you can still go this manual route.

To do so, first download the manifests we have pre-built from our corresponding repository at GitHub. We have split the whole configuration into several files to facilitate their maintenance and to provide more concise files for clearly defined purposes.

You need at least the following five files:



Create a namespace named checkmk-monitoring


Create service a account named checkmk, and a cluster role named checkmk-metrics-reader in the checkmk-monitoring namespace


Here we create the cluster collector we have named. Among other things, a service account named cluster-collector is created in the checkmk-monitoring namespace, and the service accounts are then assigned roles within the cluster. In addition, the deployment named cluster-collector will be defined.


Analogous to cluster-collector.yaml for the nodes


Create q service named cluster-collector in the checkmk-monitoring namespace. Create a service named cluster-collector-nodeport in the checkmk-monitoring namespace. The port for the NodePort is also specified here.

If you don’t want to clone the whole repository right away — which you are free to do, of course — you can use the following command to explicitly download the five files you need:

user@host:~$ URL=''; for i in 00_namespace checkmk-serviceaccount cluster-collector node-collector service; do wget "${URL}${i}.yaml"; done

If you also want to set up a network policy and a pod security policy, you will also need the following two files:

user@host:~$ URL=''; for i in network-policy pod-security-policy; do wget "${URL}${i}.yaml"; done

In the cluster-collector.yaml and node-collector.yaml files you must fill the four placeholders with concrete content. In both files you will find places where main_<YYY.MM.DD> is written. Replace these placeholders with tags from our Kubernetes collector on Docker Hub. For example, you could use the following command to replace all occurrences of main_<YYY.MM.DD> with the March 1, 2022 build tag of our container.

user@host:~$ sed -i 's/main_<YYYY.MM.DD>/main_2022.03.01/g' node-collector.yaml cluster-collector.yaml

For the external communication a service of the type NodePort is needed. This allows communication from outside and is also permanently set to TCP port 30035 in the service.yaml file. If this port is already assigned in your cluster, change the port accordingly.

Once you have made these settings, you can apply these manifest files collectively to your cluster. To do this, run the following command from the manifest location:

user@host:~$ kubectl apply -f .
namespace/checkmk-monitoring created
serviceaccount/checkmk created created created
serviceaccount/cluster-collector created created created
deployment.apps/cluster-collector created
serviceaccount/node-collector-machine-sections created
serviceaccount/node-collector-container-metrics created created
podsecuritypolicy.policy/node-collector-container-metrics-podsecuritypolicy created created
daemonset.apps/node-collector-container-metrics created
daemonset.apps/node-collector-machine-sections created
service/cluster-collector created
service/cluster-collector-nodeport created

You can also use kubectl to check whether the manifests have been applied correctly. To do this, use the following command to display all of the pods in the checkmk-monitoring namespace:

user@host:~$ kubectl get pods -n checkmk-monitoring

Furthermore, you can also check all services within the namespace as follows:

user@host:~$ kubectl get svc -n checkmk-monitoring

4. Setting up the monitoring in Checkmk

Next, in Checkmk’s GUI, we move on to setting up the special agent and a rule to automatically create hosts for your Kubernetes objects. However, to set up the special agent, there are a few prerequisites that need first to be met:

4.1. Storing password (token) in Checkmk

The best way to store the password (token) for the service account is to store it in Checkmk’s password store. This is the most secure variant, because you can separate the storage and use of the password organizationally. Alternatively, enter it directly in plain text when creating the rule (see below).

If you have kept the default checkmk-monitoring as the namespace for monitoring your Kuberentes cluster, the following command line will cut the password directly from the output from kubectl get secrets:

user@host:~$ kubectl get secret $(kubectl get serviceaccount checkmk -o=jsonpath='{.secrets[*].name}' -n checkmk-monitoring) -n checkmk-monitoring -o=jsonpath='{.data.token}' | base64 --decode

The password really is that long. If you work directly under Linux, you can add a | xsel --clipboard at the end. The password will then not be printed at all, but copied directly to the clipboard (as if you had copied it with the mouse):

user@host:~$ kubectl get secret $(kubectl get serviceaccount checkmk -o=jsonpath='{.secrets[*].name}' -n checkmk-monitoring) -n checkmk-monitoring -o=jsonpath='{.data.token}' | base64 --decode | xsel --clipboard

Add the password to the Checkmk password store with Setup > General > Passwords > Add password, for example, under the ID and title kubernetes:

kubernetes password

4.2. Importing the CA for the service account into Checkmk

In order for Checkmk to trust the Certificate Authority (CA) of the service account, you must store the CA certificate in Checkmk. You can read out the certificate — provided you have kept checkmk-monitoring as the namespace — with the following command:

user@host:~$ kubectl get secret $(kubectl get serviceaccount checkmk -o=jsonpath='{.secrets[*].name}' -n checkmk-monitoring) -n checkmk-monitoring -o=jsonpath='{\.crt}' | base64 --decode

Copy everything here including the lines BEGIN CERTIFICATE and END CERTIFIACTE and add the certificate in the Setup menu under Setup > General > Global settings > Site management > Trusted certificate authorities for SSL.

kubernetes ca

4.3. Creating a Piggyback source host

Create a new host in Checkmk in the usual way and name it mykubernetesclusterhost, for example. As the title and host name suggest, this host is used to collect the Piggyback data and also to map all services and metrics at the cluster level. Since this host only receives data via the special agent, set the IP address family option to No IP.

4.4. Setting up the dynamic host configuration

To ensure separation between the numerous Kubernetes objects and the rest of your monitoring environment, it is a good idea to first create a folder via Setup > Hosts > Add folder in which the dynamic host configuration can automatically create all required hosts. Creating or using such a folder is not required, but is a very practical option.

However, it is absolutely necessary to set up a connection for the piggyback data. Via Setup > Hosts > Dynamic host management > Add connection you get to the page for the corresponding setup. First enter a title and then click show more under Connection Properties.

Next, click Add new element and under Create hosts in select the folder you created earlier.

In a Kubernetes environment, where monitorable and monitored objects naturally come and go, it is also recommended to enable the Automatically delete hosts without piggyback data option. What exactly this option does and under what circumstances hosts are then actually deleted is explained in the section Automatically deleting hosts in the article on dynamic host configuration.

Now enter the previously created Piggyback source host under Restrict source hosts and enable the Discover services during creation option.

Once configured, the Connection Properties section of this new connector might look like the following:

Sample dynamic host configuration settings.

4.5. Setting up the special agent

Now that all the prerequisites are in place in the cluster and in Checkmk, you can turn your attention to configuring the special agent. This can be found via Setup > Agents > VM, Cloud, Container > Kubernetes.

First of all, you need to assign a name for the cluster you want to monitor. You can choose this name freely. It is used to give a unique name to all objects that originate from exactly this cluster. For example, if you enter mycluster here, the names of the hosts of all pods from this cluster will later start with pod_mycluster. The next part of the host name will then always be the namespace in which this Kubernetes object exists.

Under Token, now select the previously created entry from the password store of Checkmk.

Under API server connection > Endpoint Checkmk now requires the entry of the URL (or IP address) via which your Kubernetes API server can be reached. The port must also be specified here if the service was not provided via a virtual host. The easiest way to find out this IP address — if you don’t already have it handy — depends on your Kubernetes environment. The following command will give you the endpoint of the API server as IP address and port, which you will find as the last entry under server in the shortened output:

user@host:~$ kubectl config view
apiVersion: v1
  - cluster:
    certificate-authority-data: DATA+OMITTED
    name: my-kubernetes

If the server is provided via a DNS record, the output will look more like this instead:

user@host:~$ kubectl config view
apiVersion: v1
  - cluster:
    certificate-authority-data: DATA+OMITTED
    name: xyz:aws:eks:eu-central-1:150143619628:cluster/my-kubernetes

If you have stored the CA of your cluster — as described above — in Checkmk, you can select Verify the certificate under SSL certificate verification.

If your Kubernetes API server is only accessible via a proxy or special timeouts are required for the connection, you can enter these under HTTP proxy and TCP timeouts.

Next, you have the option to enrich the monitoring of your Kubernetes cluster with usage data collected by the Checkmk cluster collector. To do this, you need to specify the protocol, URL, and port of the cluster collector under Collector NodePort/Ingress endpoint. If you set it up using our templates, the port here is 30035 by default. If you have customized the port in the service.yaml file, change the port here accordingly. You should be able to find out the URL or IP address of the nodeport from the description of the cluster-collector pod. Just run the following command and look in the output in the line starting with Node::

user@host:~$ kubectl describe pod $(kubectl get pods --no-headers -o custom-columns="") | grep -A5 Name:.*cluster-collector
Name:         cluster-collector-5b7c8468cf-5t5hj
Namespace:    checkmk-monitoring
Priority:     0
Node:         minikube/
Start Time:   Wed, 03 Mar 2022 20:54:45 +0100
Labels:       app=cluster-collector

With the options Collect information about…​ you can now finally select which objects within your cluster should be monitored. Our preselection covers the most relevant objects. If you decide to monitor the Pods of CronJobs as well, please refer to the inline help for this point.

Last but not least, you can choose whether you want to monitor only certain namespaces within your clusters or whether explicit namespaces should be excluded from monitoring. You specify this using the Monitor namespaces option.

Your rule might now look like the following:

Example of a completed rule for a Kubernetes special agent.

Important: Under Conditions > Explicit hosts you must now re-enter the previously created host:

Rules for special agents must always be set to explicit hosts, as seen here.

Next, store the rule and perform a service discovery on this host. You will see the first cluster-level services right here:

An example image of the first service discovery after the configuration is complete.

Afterwards, activate all the changes you made and let the dynamic host configuration do the work from now on. It will generate all hosts for your Kubernetes objects after a short time.

5. Labels for Kubernetes objects

Checkmk automatically generates labels for Kubernetes objects such as clusters, deployments, or namespaces during a service discovery. All labels for Kubernetes objects that Checkmk automatically generates start with cmk/kubernetes/. For example, a pod always gets a label of the node (cmk/kubernetes/node:mynode), a label that just shows that this object is a pod (cmk/kubernetes/object:pod) and a label for the namespace (cmk/kubernetes/namespace:mynamespace). This makes it very easy to create filters and rules for all objects of the same type or in the same namespace.

6. Hardware/Software Inventory

Kubernetes monitoring by Checkmk also supports HW/SW Inventory.

An example image of hardware and software inventory of a pod.

7. Removing Checkmk

If you have deployed Checkmk to your cluster via our templates, you can remove created accounts, services and so on just as easily as with setting them up. To do this, go back to the directory where the YAML files are located and run the following command:

user@host:~$ kubectl delete -f .
namespace "checkmk-monitoring" deleted
serviceaccount "checkmk" deleted "checkmk-metrics-reader" deleted "checkmk-metrics-reader-binding" deleted
serviceaccount "cluster-collector" deleted "checkmk-cluster-collector" deleted "checkmk-token-review" deleted
deployment.apps "cluster-collector" deleted
serviceaccount "node-collector-machine-sections" deleted
serviceaccount "node-collector-container-metrics" deleted "node-collector-container-metrics-clusterrole" deleted
podsecuritypolicy.policy "node-collector-container-metrics-podsecuritypolicy" deleted "node-collector-container-metrics-cluterrolebinding" deleted
daemonset.apps "node-collector-container-metrics" deleted
daemonset.apps "node-collector-machine-sections" deleted
service "cluster-collector" deleted
service "cluster-collector-nodeport" deleted
On this page