Monitoring Google Cloud Platform (GCP)

In Checkmk, all objects to be monitored are arranged in a hierarchical structure of hosts and services. However the concept of hosts does not exist in this form in cloud-based services. To preserve the simplicity and consistency of Checkmk, we map GCP objects to our host/service schema. Each of your projects in the Google Cloud is assigned to its own host in Checkmk. All of the cloud products that you want to monitor in this project will then be split across multiple services on this dedicated host.

A small project in which only one Compute Engine VM is running can then look like this in the Checkmk monitoring:

GCP provides an HTTP-based API through which monitoring data can also be retrieved. Checkmk accesses this API via the agent_gcp special agent. This takes the place of the Checkmk agent, but — unlike the latter — is executed locally on the Checkmk server.

Log in to the Google Cloud Console. Make sure that the correct project is selected in the title bar or select the project to be monitored here.

Then open the project’s dashboard. Here you should find — if the dashboard still corresponds to the standard — a card with the Project ID. Copy or write this information down.

If the card with the project information no longer appears in your dashboard, you can also find the necessary ID via the Project settings:

Next, open the user administration under IAM & Admin. In the overview on the left-hand side, you must now select Service Accounts and then at the top click on Create Service Account. Here choose a name for the service account. We recommend giving this account a name that makes it immediately clear what it is for, e.g. checkmk-monitoring. In addition to a descriptive name, you can optionally enter a description — a service account description. After clicking on Create and continue you will need to assign the two roles, Monitoring Viewer and Cloud Asset Viewer to this service account. To do this, click in the Select a role field and enter the role’s name.

Note: If you enter Monitoring Viewer in the field, you will be shown a whole series of roles with similar names. Be careful to actually select Monitoring Viewer.

Once you have selected the roles, you can skip the next optional step and click directly on Done.

So that you can actually access the monitoring and asset data in your Google Cloud via this new service account, you will still need to create a key. You will store this key later in the corresponding rule in Checkmk or in the Password store.

In the Service accounts for project My Project overview, you can click on the three dots in the row by your new service account and select Manage keys. Next, click on Add key and then on Create new key. Be sure to select JSON as the format and click Create. This click on Create — easily overlooked — downloads a file in JSON format. Keep this file in a safe place for the time being, as you will not be able to download it again. We do recommend, however, that you also delete this file after storing its contents in Checkmk (see Creating a rule for GCP agents). If necessary, a new key should be created and the old one discarded altogether.

On your GCP project’s overview page you will also find the APIs & Services menu item. In this overview, check whether the Cloud Asset API appears in the list of Enabled APIs & services. If this is not the case, activate this API via the Enable APIs and services button. Following an activation, it will take a few minutes until the API is actually accessible.

In the Google Cloud Platform, billing information is stored separately from the resources. As a rule, separate projects are created for cost analysis in GCP, in which the billing information for other projects is also collected. In order to be able to monitor this information with Checkmk, it is essential that this data is exported to BigQuery within GCP. Only data that is available in BigQuery tables can be accessed remotely — and thus by Checkmk. How to set up such an export within GCP is explained in detail in the document Export Cloud Billing data to BigQuery in the GCP help pages.

If you have set up BigQuery or are already using it, you will find a list of its included tables in the accounting project’s SQL workspace. Open the accounting project’s table and click on the Details tab. Under Table ID, you will find the information you will need to enter when creating the rule in Checkmk at Costs > BigQuery table ID.

The service for monitoring project costs is designed as an overview. Only the monthly costs of the individual projects are displayed and monitored. You can define threshold values for these monthly costs with the GCP Cost rule.

Now create a host for monitoring GCP in Checkmk. You can assign the host name as you wish. If you want to monitor more than one project in GCP, you must create a separate host in Checkmk for each project.

Important: Since GCP as a service has neither an IP address nor a DNS name (the special agent does the access by itself), you must set the IP address family to No IP.

As mentioned at the beginning of this article, projects on the Google Cloud Platform are monitored by a special agent. This agent is configured with a rule, which you can find via Setup > Agents > VM, cloud, container > Google Cloud Platform (GCP).

In the corresponding field, enter the Project ID that you previously looked up in your project.

Under JSON credentials for service account, the key that you earlier created for your service account must be entered next. You will need to copy the entire JSON object (including the curly brackets) into this.

Under GCP services to monitor you can now select which GCP products are to be monitored by the special agent. To make the API queries as economical as possible, we recommend that you only select the products that are actually being used in your project.

Now start a service discovery of the newly created GCP host in which Checkmk should now find quite a few services. When you have added the services, it will look something like this after activating the changes in the monitoring:

Services that are assigned to Compute Engine VM instances are not assigned to the GCP host, but to so-called piggyback hosts. This works in such a way that data retrieved from the GCP host is distributed to these piggyback hosts, which operate without their own monitoring agents. A piggyback host is assigned to each VM instance. The names of these piggyback hosts are composed from the ID of your project, an underscore and the full name of the instance. For example, if your project has the project ID my-project-19001216 and you then monitor a VM with the name my-instance01, the piggyback host will be named my-project-19001216_my-instance01. Either create these hosts manually or — if possible — leave this task to the dynamic host management.

The Exceptions service supports you in setting up a GCP monitoring, and also assists with any future problems encountered when communicating with the Google Cloud API. All error messages that the Google Cloud API returns to the special agent are collected and processed here. In the event of an error, this service becomes CRIT by default and in its Summary will provide an indication of where the problem lies.

Clicking on the name of the service will then give you a very detailed message, often with a link to the exact place in your GCP project where it needs a different setting, for instance. In the following example, the Cloud Asset API is disabled in the monitored project.

A click on the WWW globe will then take you to the exact page in your project where this API can be enabled.