Monitoring Kubernetes

For an introduction to the new monitoring of Kubernetes, we recommend our two videos Kubernetes Monitoring with Checkmk and Detecting issues and configuring alerts for Kubernetes clusters.

Since Kubernetes clusters can quickly undergo major changes in terms of the number and location of individual components, we recommend creating a separate site for monitoring your Kubernetes environment. You can then connect this site to your central site as usual via distributed monitoring.

Checkmk monitors your Kubernetes clusters in two ways:

The Kubernetes special agent simply retrieves basic information via your cluster’s API server. This can already be used to retrieve the states of nodes and containers. Most of the metadata of your pods and deployments are also obtained in this way.

For a comprehensive monitoring, however, something is still missing up to this point. Questions such as how much load a particular deployment generates on the CPU, or how much memory a DaemonSet is currently using cannot be answered in this way.

This is where our Checkmk Node Collector and our Checkmk Cluster Collector come into play. These are an indispensable part of Kubernetes monitoring within Checkmk. A not insignificant part of what follows in this article is thus also about installing and setting these up. In addition, the use of the Kubernetes dashboards in the Checkmk Enterprise Editions only makes sense if the Node and Cluster Collectors can provide data on the loads for this.

When monitoring pods and replicas in your Kubernetes clusters, status changes or delays sometimes occur much more frequently. To account for this, the checks for certain states of these objects only change their status in Checkmk after 10 minutes.

Kubernetes monitoring in Checkmk has been rewritten from scratch. The scope of the data that can be monitored has grown considerably. Since the technical basis for Kubernetes monitoring in Checkmk 2.1.0 is fundamentally different, it is not possible to transfer or even rewrite previous monitoring data from your Kubernetes objects.

The installation of Kubernetes monitoring is done with the help of the helm tool. Helm is also suitable for less experienced users and standardizes the management of configurations. Helm is a kind of package manager for Kubernetes. If you are not yet using Helm, you can usually get it from your Linux distribution’s package manager or from the Helm project website.

You can use Helm to include repositories as sources and easily add the Helm charts that these contain to your cluster in the same way as packages. First of all, identify the repository. In the following example, we use the name tribe29 to make it easier to access the repository later. You can, of course, also use any other name of your choice:

We update our Helm charts whenever new developments in Kubernetes require it. It is therefore worth checking from time to time whether new versions are available in the repository. If you have named your local copy of our repository tribe29, as in the previous command, you can use the following command to display all versions of the charts available in the repository:

If a newer version is available you can update with helm repo update.

Since we cannot know in advance how your Kubernetes cluster is structured, we have chosen the safest variant for how the Cluster Collectors are started: By default, you do not provide any ports that can be accessed remotely. In order to be able to access the collectors later, you will need to adapt these settings to your particular cluster.

We support two communication paths by default: the query via Ingress and the query via NodePort. The configuration of these will vary depending on which variant you support in your cluster.

In order to be able to determine certain parameters yourself across all configurations, you include a control file, the so-called values.yaml.

There are two ways to create such a values.yaml. You can either extract the file supplied by us in the Helm charts and edit it, or you can simply create a minimal version yourself.

Whenever you want to deploy changes to this file in your cluster, you can again use the Helm chart installation command that we will cover later in this article.

After customizing values.yaml or creating your own, use the following command to install all of the necessary components into your cluster to be able to monitor it in Checkmk:

Since this command is not self-explanatory, we provide an explanation of the individual options below:

Once you have executed the command, your Kubernetes cluster is prepared for monitoring with Checkmk. The cluster will now take care of itself to ensure that the necessary pods and the containers they contain are running and accessible.

It is best to store the password (token) for the service account in the Checkmk password store. This is the safest option, as you can separate the storage and use of the password organizationally. Alternatively, enter the password directly in plain text when creating the rule (see below). For information on how to display the required password, see the Helm chart output. Add the password to the Checkmk password store with Setup > General > Passwords > Add password e.g. under the ID and the title My Kubernetes Token:

In order for Checkmk to trust the service account’s Certificate Authority (CA), you must store the CA certificate in Checkmk. How to display the required certificate is also in the Helm chart output. Copy everything here including the lines BEGIN CERTIFICATE and END CERTIFICATE and add the certificate in the Setup menu under Setup > General > Global settings > Site management > Trusted certificate authorities for SSL:

Create a new host in Checkmk in the usual way and name it mykubernetesclusterhost for example. As the title and host name suggest, this host is used to collect the piggyback data and also to map all services and metrics at the cluster level. Since this host only receives data via the special agent, be sure to set the IP address family option to No IP in the host’s properties.

To ensure separation between the objects of different Kubernetes clusters, it can be useful to create a folder per cluster via Setup > Hosts > Add folder, in which the dynamic host configuration can automatically create all hosts of a cluster. However, creating or using such a folder is optional.

Next, set up a connector in the Checkmk Enterprise Editions for the incoming piggyback data: with Setup > Hosts > Dynamic host management > Add connection. First enter a title and then click on show more under Connection Properties.

Next, click Add new element and select the previously created folder under Create hosts in.

Leave the default attributes under Host attributes to set as they are. They ensure that Checkmk only adheres to the piggyback data for the automatically created hosts and does not attempt to ping them or reach them via SNMP, for example.

In a Kubernetes environment where monitorable and monitored objects come and go continuously, it is also recommended to activate the option Automatically delete hosts without piggyback data. What exactly this option does and under which circumstances hosts are then actually deleted is explained in the chapter Automatic deletion of hosts in the article on dynamic host configuration.

Now enter the previously created Piggyback source host under Restrict source hosts and activate the option Discover services during creation.

The Connection Properties section of this new connector could then look like this:

In the Raw Edition of Checkmk you will need to create the hosts for the accumulating piggyback data manually. Because a large number of piggyback hosts are likely to arise here in a Kubernetes cluster, we recommend using our find_piggy_orphans script in your Checkmk site’s ~/share/doc/check_mk/treasures/ directory.

By default, Checkmk performs a service discovery every two hours and displays the result from this discovery in the Check_MK Discovery service. You can find this setting in the Periodic service discovery rule set. In the context of Kubernetes, we recommend creating a rule for all hosts with the label cmk/kubernetes:yes. This label is automatically given by Checkmk to every host that represents Kubernetes objects. You should select a shorter interval for the service discovery here, and also activate the Automatically update service configuration option. The settings in the following screenshot are only examples. You will need to decide what makes sense for your clusters on a case-by-case basis.

To restrict this rule to all of your cluster’s hosts, it is sufficient to enter cmk/kubernetes:yes in the Conditions under Host labels. However, if you want to create individual rules for multiple clusters, simply use the respective cluster-specific label here. These labels always have the form cmk/kubernetes/cluster:mycluster.

Now that all the prerequisites have been created in the cluster and in Checkmk, you can turn your attention to the configuration of the special agent. You can find this via Setup > Agents > VM, Cloud, Container > Kubernetes. Create a new rule with Add rule.

First of all, you must assign a name for the cluster to be monitored. You can choose this name freely. It is used to give a unique name to all objects that come from this particular cluster. For example, if you enter mycluster here, the names of the hosts of all pods from this cluster will later begin with pod_mycluster. The next part of the host name will then always be the namespace in which this Kubernetes object exists. The host name of a pod could then be pod_mycluster_kube-system_svclb-traefik-8bgw7, for example.

Under Token, now select the previously created entry from the Checkmk password store.

Under API server connection > Endpoint, Checkmk now asks you to enter the URL (or IP address) through which your Kubernetes API server can be accessed. You only need to enter the port if the service is not provided via a virtual host. The easiest way to find out this address — if you don’t already have it at hand — will depend on your Kubernetes environment. The following command will give you the API server’s endpoint in the server line:

However, the actual output of kubectl config view varies greatly. If a port is also specified here in the server line, be sure to include it in the rule as well.

If you have so far followed these instructions step by step and have deposited your cluster’s CA certificate — as described above — in Checkmk, select under SSL certificate verification the Verify the certificate entry.

Next, you have the opportunity to enrich the monitoring of your Kubernetes cluster with usage data collected by the Checkmk Cluster Collector. We repeat it once more here to emphasize the importance: The setup of the Cluster Collector is absolutely essential for a fully comprehensive monitoring of your clusters. This is the only way to obtain important data such as CPU and memory utilization and to receive information on the file systems used by the individual components.

So activate the Enrich with usage data from Checkmk Cluster Collector option and specify the endpoint of the NodePort or Ingress. How to re-display this endpoint is in the Helm chart output.

With the Collect information about…​ options, you can now select which objects within your cluster are to be monitored. Our preselection covers the most relevant objects. If you decide to also monitor the Pods of CronJobs, please refer to the inline help on this point.

With the next two options, you can further limit the objects to be monitored. If you are only interested in objects from certain namespaces, set this accordingly under Monitor namespaces. Here you can either enter individual namespaces to be monitored or explicitly exclude individual namespaces from monitoring.

With the Cluster resource aggregation option, you can specify nodes that do not provide resources for the workload of your cluster. These nodes should be excluded from the calculation of available resources, otherwise there is a risk that capacity bottlenecks will not be detected. By default, we therefore exclude the control-plane and infra nodes from the evaluation.

As a final option, you can import the so-called annotations from Kubernetes. In Checkmk, these annotations become host labels and can thus be used as conditions in rules. You can specify which annotations are to be imported using regular expressions. Again, consult the detailed inline help at this point.

Note: The Import all valid annotations option is provided here only for completeness. We do not recommend blindly importing all annotations, as this can create a very large mountain of useless labels in Checkmk.

Important: Under Conditions > Explicit hosts you must now enter the previously-created host:

Then save the rule and perform a service discovery on this host. You will immediately see the first cluster-level services here:

Now activate all of the changes you have made and let the dynamic host configuration do the work for you. This will create all hosts for your Kubernetes objects within a short space of time.

The Checkmk Enterprise Editions are supplied with six built-in dashboards for Kubernetes. In order to use these dashboards in a practical way, it is necessary to install and configure our Cluster Collector. Specifically, these six dashboards are called:

The entry point is always the Kubernetes dashboard, which you can access via Monitor > Applications > Kubernetes:

In the Kubernetes dashboard, all of your Kubernetes clusters being monitored will be listed on the left-hand side. This listing of clusters is also your entry point to drill deeper into Kubernetes dashboards. Clicking on the name of a cluster will take you to the selected cluster’s Kubernetes Cluster dashboard. In the Kubernetes Cluster dashboard, clicking on the respective name then takes you to the other context-dependent dashboards:

The Checkmk Kubernetes monitoring also supports the HW/SW inventory. For example, if you click on the cluster’s primary name (here: mycluster) in the above cluster dashboard, you will jump to the cluster’s inventory.

In the same way, i.e. via the boxes with the primary names of the objects, you will also get to the inventory for the respective object in the other dashboards. The following example shows the HW/SW inventory from a pod: