Monitoring OpenShift

Since OpenShift clusters can also undergo major changes very quickly in terms of the number and location of individual components, we recommend creating a separate Checkmk site for monitoring an OpenShift environment. This can then be connected to the central site as usual via the distributed monitoring procedure.

Checkmk monitors your OpenShift clusters in two ways:

First, you need to set up a namespace and service account for Checkmk in your OpenShift cluster. The quickest way to do this is via the OpenShift CLI (oc for short).

In the following example, we will name this namespace checkmk-monitoring. If you want or need to choose a different name, you will also need to make this change when creating the service account.

The service account with its associated role and the so-called RoleBinding can be created by specifying our ready-made YAML file published on GitHub. Check its contents and then run the following command:

Alternatively, you can first download this YAML file, customize it according to your needs, and then apply oc apply -f to this local copy of the file.

With the oc command line tool you can now also read out all of the information from your cluster, which you will usually have to specify to set up the special agent. If you have changed the service account name or namespace, you must edit this information as described for the following commands.

It is best to store the service account’s password (token) in the Checkmk password store. This is the safest option, since you can separate the storage and use of the password in an organized way. Alternatively, enter the password directly in plain text when creating the rule (see below).

To add the password to the Checkmk password store, navigate to Setup > General > Passwords > Add password. We use My OpenShift Token as the ID and title in our example:

In order for Checkmk to trust the service account’s certificate chain, you will need to store it in Checkmk. Copy everything here — including the lines containing BEGIN CERTIFICATE and END CERTIFICATE — and add the certificate in the Setup menu under Setup > General > Global settings > Site management > Trusted certificate authorities for SSL:

Create a new host in Checkmk in the usual way and name it myopenshiftclusterhost for example. As the title and host name suggest, this host is used to collect the piggyback data and also to map all services and metrics at the cluster level. Since this host receives data exclusively through the special agent, in the host’s properties be sure to set the IP address family option to No IP. Confirm this whole configuration by pressing the Save & view folder button.

To ensure separation between objects in different Kubernetes clusters, it is usually convenient to create a folder per cluster via Setup > Hosts > Add folder, in which the dynamic host configuration can create all of a cluster’s hosts automatically. However, creating and using such a folder is optional.

Next, set up a connector for the incoming piggyback data. To do this, navigate to Setup > Hosts > Dynamic host management > Add connection. First enter a title and then under Connection Properties click show more.

The next very important step is to enter the previously created piggyback source host under Restrict source hosts.

Then, under Piggyback creation options click Add new element and under Create hosts in select the previously created folder.

You can leave the default attributes under Host attributes to set as they are. These ensure that Checkmk only accepts the piggyback data from the automatically created hosts and does not attempt to ping these or access them via SNMP, for example.

In an OpenShift environment where monitorable and monitored objects continuously come and go, it is usually recommended to also enable the Automatically delete hosts without piggyback data option. What exactly this option does and under which circumstances hosts are then actually deleted is explained in the automatic deletion of hosts chapter in the article on dynamic host configuration.

Finally, enable the Discover services during creation option.

The Connection Properties section of this new connector might look like the following:

By default, Checkmk performs a service discovery every two hours and displays the result from this discovery in the Check_MK Discovery service. You can find this setting in the Periodic service discovery rule set. In the context of OpenShift, we recommend creating a rule with the label cmk/kubernetes:yes for all hosts. This is because every host representing Kubernetes objects automatically receives this label from Checkmk. In this case you should choose a shorter interval than two hours for the service discovery, and also activate the Automatically update service configuration option. The settings in the below screenshot are only examples. You will need to decide what makes sense for your clusters on a case-by-case basis.

To restrict this rule to all hosts in your cluster, under Host labels simply enter cmk/kubernetes:yes in the Conditions. However, if you also want to create different rules for different clusters, just use the respective cluster-specific label here. These labels are always in the form cmk/kubernetes/cluster:mycluster.

Now that all prerequisites have been created in the cluster and in Checkmk, you can turn your attention to the special agent configuration. This can be found via Setup > Agents > VM, cloud, container > Kubernetes. Create a new rule with Add rule.

First of all, you must assign a name for the cluster to be monitored. This name can be specified freely as desired. It is used to assign a unique name to all objects that originate from this specific cluster. For example, if you enter mycluster here, the names of the hosts for all of the pods in this cluster will later begin with pod_mycluster. The next part of the host name will then always be the namespace in which this Kubernetes object exists. For example, the host name of a pod could then be pod_mycluster_kube-system_svclb-traefik-8bgw7.

Under Token, now select the previously created entry from the Checkmk password store.

Under API server connection > Endpoint, Checkmk now requires you to enter the URL through which your Kubernetes API server can be contacted. The port is only required if the service is not provided via a virtual host. Here, enter the address that you determined in the Retrieve the Kubernetes API endpoint section.

If you have so far followed these instructions step by step and have — as described above — deposited your cluster’s CA certificate in Checkmk, then under SSL certificate verification select Verify the certificate.

Enable the Enrich with usage data option, select Use data from OpenShift in the following menu, and enter the Prometheus API endpoint that you determined in the Retrieve the Prometheus API endpoint section.

The list under Collect information about…​ allows you to select which objects within your cluster should be monitored. This list covers the most relevant objects. If you also decide to monitor the Pods of CronJobs, please refer to the inline help for this option.

The next two selections allow you to further narrow down the objects to be monitored. If you are only interested in the objects from certain namespaces, set this accordingly under Monitor namespaces. Here you can either enter individual namespaces to be monitored or explicitly exclude individual namespaces from monitoring.

The Cluster resource aggregation option allows you to designate nodes that do not provide resources for your cluster’s workload. These nodes should be excluded from the calculation of available resources. Otherwise, there is a risk that capacity bottlenecks will not be detected. We therefore by default exclude the control-plane and infra nodes from the calculation.

As a final option, you can import so-called annotations from Kubernetes. In Checkmk, these annotations become host labels and can thus be further used as conditions in rules. Use regular expressions to specify which annotations should be imported. Consult the detailed inline help again at this point.

Note: The Import all valid annotations option is provided here only for completeness. We do not recommend importing all annotations at once, because this can create a very large mountain of useless labels in Checkmk.

Important: Under Conditions > Explicit hosts you must now enter the previously created host:

Then save the rule and perform a service discovery for this host. The first cluster-level services will appear right away here:

Now activate all of the changes you have made and let the dynamic host configuration do the work from now on. This will generate all of the hosts for your Kubernetes objects within a short period of time.

The Checkmk commercial editions are supplied with six built-in dashboards for Kubernetes. In order to make sense of these dashboards, it is necessary to have our Cluster Collector installed and configured. Specifically, these dashboards are called:

The entry point is always the Kubernetes dashboard, which you can access via Monitor > Applications > Kubernetes:

In the Kubernetes dashboard, all of your monitored clusters are listed on the left side. This listing of clusters is also your entry point for drilling deeper into the dashboards. Clicking on the name of a cluster will take you to the Kubernetes Cluster dashboard for the selected cluster. In the Kubernetes Cluster dashboard, clicking on the respective name then takes you to the other context-dependent dashboards:

Monitoring OpenShift with Checkmk also supports HW/SW inventory. For example, in the cluster dashboard above, clicking on the cluster’s ID name (here: mycluster) will take you to the cluster’s inventory.

In the same way, i.e. in the other dashboards via the boxes with the ID names of the objects, the inventory for each respective object can be displayed. The following example shows the HW/SW inventory for a pod:

If you have used our default YAML file to create the service account, you can also delete it as follows: