Monitoring Kubernetes

For an introduction to the new monitoring of Kubernetes, we recommend the two videos Kubernetes Monitoring with Checkmk and Detecting issues and configuring alerts for Kubernetes clusters.

Kubernetes monitoring in Checkmk has been rewritten from scratch. The volume of data that can be monitored has significantly increased. Since the technical basis of Kubernetes monitoring is fundamentally different in Checkmk 2.1.0, it is not possible to adopt or even rewrite previous monitoring data for your Kubernetes objects.

We currently recommend installing Kubernetes monitoring using the helm tool, as it is also suitable for less experienced users and standardizes the management of configurations. Helm is a kind of a package manager for Kubernetes. You can use it to include repositories as sources and easily add the Helm charts they contain like packages to your cluster. To do this, first of all make the repository known. In the following example, we use the name tribe29 to make it easier to access the repository later. However, you can of course use any other name of your choice:

Using Helm, you create the necessary configuration files completely independently. In order to be able to determine certain parameters over all configurations, you specify a control file, the so-called values.yaml. As a starting point we recommend the template provided by us. Copy it and adapt it to your own environment.

Since we cannot know in advance how your Kubernetes cluster is set up, we have chosen the safest option for how the Checkmk collectors are started: By default, they do not expose any ports to be reached from the outside. To allow you to access the collectors later, adjust these settings accordingly.

For simplicity, let’s take our template as a starting point. We support two communication paths by default: the query via Ingress and the query via NodePort. Depending on which variant you support in your cluster, the configuration will vary.

After customizing values.yaml or creating your own files, use the following command to create all of the necessary configuration files to set up your Kubernetes cluster for monitoring in Checkmk:

Since the command is not self-explanatory, we will provide an explanation of each option below:

After you run the command, your Kubernetes cluster will be ready to be monitored with Checkmk. The cluster will now take care of itself to ensure that the necessary pods and the containers within them are running and accessible.

It does not normally make sense for you to customize the manifests (configuration files) yourself. On the one hand, because you need detailed knowledge of the architecture of the Checkmk Kubernetes Collectors for this and, on the other hand, because manual customization is much more error-prone. For example, in helm you set up communication over TLS once only, instead of needing to add it to all relevant places in each actual individual manifest.

However, if you don`t use helm, or want to have control over all the setup details, you can still go this manual route.

To do so, first download the manifests we have pre-built from our corresponding repository at GitHub. We have split the whole configuration into several files to facilitate their maintenance and to provide more concise files for clearly defined purposes.

You need at least the following five files:

If you don’t want to clone the whole repository right away — which you are free to do, of course — you can use the following command to explicitly download the five files you need:

If you also want to set up a network policy and a pod security policy, you will also need the following two files:

In the cluster-collector.yaml and node-collector.yaml files you must fill the four placeholders with concrete content. In both files you will find places where main_<YYY.MM.DD> is written. Replace these placeholders with tags from our Kubernetes collector on Docker Hub. For example, you could use the following command to replace all occurrences of main_<YYY.MM.DD> with the March 1, 2022 build tag of our container.

For the external communication a service of the type NodePort is needed. This allows communication from outside and is also permanently set to TCP port 30035 in the service.yaml file. If this port is already assigned in your cluster, change the port accordingly.

Once you have made these settings, you can apply these manifest files collectively to your cluster. To do this, run the following command from the manifest location:

You can also use kubectl to check whether the manifests have been applied correctly. To do this, use the following command to display all of the pods in the checkmk-monitoring namespace:

Furthermore, you can also check all services within the namespace as follows:

The best way to store the password (token) for the service account is to store it in Checkmk’s password store. This is the most secure variant, because you can separate the storage and use of the password organizationally. Alternatively, enter it directly in plain text when creating the rule (see below).

If you have kept the default checkmk-monitoring as the namespace for monitoring your Kuberentes cluster, the following command line will cut the password directly from the output from kubectl get secrets:

The password really is that long. If you work directly under Linux, you can add a | xsel --clipboard at the end. The password will then not be printed at all, but copied directly to the clipboard (as if you had copied it with the mouse):

Add the password to the Checkmk password store with Setup > General > Passwords > Add password, for example, under the ID and title kubernetes:

In order for Checkmk to trust the Certificate Authority (CA) of the service account, you must store the CA certificate in Checkmk. You can read out the certificate — provided you have kept checkmk-monitoring as the namespace — with the following command:

Copy everything here including the lines BEGIN CERTIFICATE and END CERTIFIACTE and add the certificate in the Setup menu under Setup > General > Global settings > Site management > Trusted certificate authorities for SSL.

Create a new host in Checkmk in the usual way and name it mykubernetesclusterhost, for example. As the title and host name suggest, this host is used to collect the Piggyback data and also to map all services and metrics at the cluster level. Since this host only receives data via the special agent, set the IP address family option to No IP.

To ensure separation between the numerous Kubernetes objects and the rest of your monitoring environment, it is a good idea to first create a folder via Setup > Hosts > Add folder in which the dynamic host configuration can automatically create all required hosts. Creating or using such a folder is not required, but is a very practical option.

However, it is absolutely necessary to set up a connection for the piggyback data. Via Setup > Hosts > Dynamic host management > Add connection you get to the page for the corresponding setup. First enter a title and then click show more under Connection Properties.

Next, click Add new element and under Create hosts in select the folder you created earlier.

In a Kubernetes environment, where monitorable and monitored objects naturally come and go, it is also recommended to enable the Automatically delete hosts without piggyback data option. What exactly this option does and under what circumstances hosts are then actually deleted is explained in the section Automatically deleting hosts in the article on dynamic host configuration.

Now enter the previously created Piggyback source host under Restrict source hosts and enable the Discover services during creation option.

Once configured, the Connection Properties section of this new connector might look like the following:

Now that all the prerequisites are in place in the cluster and in Checkmk, you can turn your attention to configuring the special agent. This can be found via Setup > Agents > VM, Cloud, Container > Kubernetes.

First of all, you need to assign a name for the cluster you want to monitor. You can choose this name freely. It is used to give a unique name to all objects that originate from exactly this cluster. For example, if you enter mycluster here, the names of the hosts of all pods from this cluster will later start with pod_mycluster. The next part of the host name will then always be the namespace in which this Kubernetes object exists.

Under Token, now select the previously created entry from the password store of Checkmk.

Under API server connection > Endpoint Checkmk now requires the entry of the URL (or IP address) via which your Kubernetes API server can be reached. The port must also be specified here if the service was not provided via a virtual host. The easiest way to find out this IP address — if you don’t already have it handy — depends on your Kubernetes environment. The following command will give you the endpoint of the API server as IP address and port, which you will find as the last entry under server in the shortened output:

If the server is provided via a DNS record, the output will look more like this instead:

If you have stored the CA of your cluster — as described above — in Checkmk, you can select Verify the certificate under SSL certificate verification.

If your Kubernetes API server is only accessible via a proxy or special timeouts are required for the connection, you can enter these under HTTP proxy and TCP timeouts.

Next, you have the option to enrich the monitoring of your Kubernetes cluster with usage data collected by the Checkmk cluster collector. To do this, you need to specify the protocol, URL, and port of the cluster collector under Collector NodePort/Ingress endpoint. If you set it up using our templates, the port here is 30035 by default. If you have customized the port in the service.yaml file, change the port here accordingly. You should be able to find out the URL or IP address of the nodeport from the description of the cluster-collector pod. Just run the following command and look in the output in the line starting with Node::

With the options Collect information about…​ you can now finally select which objects within your cluster should be monitored. Our preselection covers the most relevant objects. If you decide to monitor the Pods of CronJobs as well, please refer to the inline help for this point.

Last but not least, you can choose whether you want to monitor only certain namespaces within your clusters or whether explicit namespaces should be excluded from monitoring. You specify this using the Monitor namespaces option.

Your rule might now look like the following:

Important: Under Conditions > Explicit hosts you must now re-enter the previously created host:

Next, store the rule and perform a service discovery on this host. You will see the first cluster-level services right here:

Afterwards, activate all the changes you made and let the dynamic host configuration do the work from now on. It will generate all hosts for your Kubernetes objects after a short time.