Understanding and configuring services

Once you have added a new host in the setup the next step is to call up the list of services. With this action the automatic service discovery takes place for the first time for this host, and the accessibility of the data sources is checked in parallel. You can also call up this list at any time later in order to restart the discovery or to carry out modifications to the configuration. There are various ways of opening the service list:

When a host has been newly-incorporated its services have not yet been configured, and therefore all discovered services appear in the Undecided services - currently not monitored category:

The usual method for adding the newly discovered services is to simply hit the Accept all button. That way also all host labels will be added in one go. Afterwards quickly Activate changes — after which the host will be in the monitoring.

For a host that is already being monitored this list looks different. Instead of Undecided services - currently not monitored you will see Monitored services. If Checkmk detects something on a host that is not being monitored, that however should be monitored, then the list will look something like this:

A click on Monitor undecided services or on Accept all simply adds all of the missing services so that the monitoring is again complete. If you only want to add some of the missing services, you can alternatively click the button in order to Move to monitored services.

In data centers things can not only newly appear, but also disappear. A data base instance can be discontinued, a LUN unmounted, a file system removed, etc. Checkmk automatically recognizes such services as vanished. In the service list e.g., it will look like this:

The simplest way to be free of these services is again with a click on Accept all or by clicking the button in each individual line, which stands for Remove vanished services. Attention: The reason for the disappearance can of course be due to a problem! The disappearance of a file system can also mean that due to an error it could not be mounted. The monitoring is after all there for such cases! You should only remove the service when you really know that this no longer needs monitoring.

You won’t necessarily want to monitor everything that Checkmk finds. The discovery works in a target-oriented way of course, and it can exclude much unnecessary data in advance. Nonetheless, how can Checkmk know, for example, that a particular data base instance has been set up only ‘to play around with’, and is not in production? There are two ways of eliminating such services:

There are a number of plug-ins that notice things during a discovery. For example, the plug-in for network interfaces checks the speed set on the interface during the discovery. Why? In order to be able to warn you in case it changes! It is rarely a good sign when an interface is sometimes set to 10 Mbit/s, and sometimes to 1 Gbit/s — this could rather be an indication of a defective auto-negotiation.

What happens when this change is desired and is to be accepted as OK from now on? Either — remove the service via icon , which stands for Move to undecided services and re-add it immediately afterwards. Or refresh all services of the host by clicking Actions > Remove all and find new in the menu bar. This is naturally much easier, but only when you don’t want to keep individual services in an error state.

Updating only the labels can be easily achieved via Actions > Host labels > Update host labels and Actions > Services > Update service labels respectively in the menu bar. If it is only necessary to add or update individual (new) service labels, this can be done in the line of the respective service in the Changed services box by clicking on .

There are a few special features for devices that are monitored via SNMP. You can learn about these in the Article about SNMP.

It would make little sense for Checkmk to simply define a service to monitor every process found on a host. Most processes are either of no interest or are only temporarily present. At the very least there are hundreds of processes running on a typical Linux server.

For monitoring services you therefore need to work with enforced services or — and this is much more elegant — by using the rule set Process discovery to tell the service discovery which processes it should be on the lookout for. In this manner you can always allow a monitoring to be instituted automatically when a definitely interesting process is found on a host.

The following image shows a rule in the Process discovery rule set which searches for processes that execute the program /usr/sbin/apache2. In this example a service (Grab user from found processes) will be created for every different operating system user for whom such a process is found. The service will be named Apache %u, where %u will be replaced by the user name. For the threshold the number of process instances will be set to 1/1 (minimum) and 30/60 (maximum) respectively:

Please note that the predefined thresholds are referred to as Default parameters for detected services. You can assign these — and likewise all other services — via rules. As a reminder: the above rules configure the service discovery — the what. If the services are present for the first time, the rule chain State and count of processes is responsible for the thresholds.

The fact that you can set thresholds during a discovery is an aid to convenience. There is a catch though: changes to the discovery rule only take effect with the next discovery. If you change thresholds you will need to run a new discovery. If, however, you only use the rule to discover the services (the what), and the rule set State and count of processes for the how, then you will not have this problem.

In order to monitor certain or single processes on a Windows host, you just have to state the filename (incl. the extension) without any path in the field Executable. You can find all these names in the details tab of the Windows Task Manager for example. In the rule Process discovery this could look like this for the process svchost:

Further information on process discovery can be found in the inline help for this rule set. As always you can activate this via Help > Show inline help in the menu bar.

The discovery and parameterizing of the monitoring of Windows services is analogous to the processes and is controlled via the rule sets Windows service discovery (what) and Windows services (how) respectively. Here is an example of a rule that watches out for two services:

Exactly as for the processes, here the service discovery is also only one option. If, on the basis of host characteristics and folders, you can formulate precise rules for hosts on which specific services are to be expected, then you can also work with enforced services. This is independent of the situation actually found — it can however require considerably more effort, as under these circumstances you need many rules in order to exactly describe which service is to be expected on which host.

Checkmk uses the same logic for monitoring network interfaces on servers and ports on Ethernet switches. With switch ports the existing options for controlling the service discovery are especially interesting, even though (in contrast to the processes and Windows services) the discovery initially functions without rules. That is to say, by default Checkmk automatically monitors all physical ports that currently have an UP state. The applicable rule set is called Network interface and switch port discovery and offers numerous setting options that are only briefly described here:

The following options are the most important ones:

Much better for this however is a regular discovery check, which is set up automatically on new sites. This service Check_MK Discovery exists for every host and will log a warning whenever it finds unmonitored items:

The details of unmonitored or vanished services can be found on the Check_MK Discovery service details page in the Details field:

The host’s service list in the Setup can be easily accessed via the action menu of the Check_MK Discovery service and the Run service discovery entry.

The parameterizing of the discovery check is very simply done using the Periodic service discovery rule set. In a site which is fresh out the box, you will already find one rule which has been defined to apply to all hosts:

Alongside the interval in which the check is to be run, you can set monitoring state for cases of unmonitored services, vanished services, changed service labels and new host labels.

You can have the discovery check add missing services automatically. To this end activate the Automatically update service configuration option, which will make further options available.

Alongside the additions, in Parameters you can also choose to delete vanished services, or even to delete all existing services and perform a complete new discovery. The latter option can be found in the drop-down menu under the name Refresh all services and host labels (tabula rasa). Both these options should be used carefully! A vanished service can indicate a problem! The discovery check will simply delete such a service and lull you into thinking everything is in order. The refresh is especially risky. For example, the check for switchports will only take ports that are UP into the monitoring. Ports with a status of DOWN will be perceived as vanished and quickly deleted from the discovery check!

A further problem needs to be considered: adding services or even the automatic Activate Changes can distract you — the admin — when you are performing a configuration. It can theoretically occur that while you are working on rules and settings, in that moment a discovery check activates your changes. In Checkmk only all changes can be activated at once! In order to preclude such situations you can reschedule the time for this function, to overnight for example. The above image shows an example of this.

Under Vanished clustered services you can handle clustered services separately. The special feature here: If a clustered service moves from one node to another, it could be briefly considered as vanished and accordingly deleted unintentionally. If you waive this option, in turn, services that have actually disappeared would never be deleted.

The Group discovery and activation for up to setting ensures that not every single service that has been newly-found immediately triggers an Activate Changes — rather there will be a specified waiting time so that multiple changes can be activated in a single action. Even if the discovery check is set to an interval of two hours or more, this only applies to each host separately. The checks don’t run simultaneously for every host — which is a good thing, as a discovery check requires significantly more resources than a normal check.

To recap — all options at a glance:

The result of a service discovery — thus, as explained earlier, the tables of host names, check plug-ins, items and identified parameters — can be found in the var/check_mk/autochecks folder. Here, for every host there is a data set that stores the automatically-discovered services. As long as you don’t damage this data set’s Python syntax you can alter or delete individual lines manually. Deleting the data set removes all services and flags them as quasi ‘unmonitored’ again.

So far you have learned how to include services in monitoring. Now it makes little sense to have to look at lists of thousands of services and/or always have to go through host views. For example, if you want to view all file system or update services together, you can simply assemble groups in a similar way as you can with host groups.

Service groups make it easy for you to bring a lot more order to monitoring via views and NagVis maps, and to switch targeted notifications and alert handlers. By the way – you could almost always construct corresponding views purely using the view filters – but service groups are more clearly arranged and easier to work with.

You find the service groups via Setup > Services > Service groups.

Creating a service group is simple: Create a group via Add group and assign a name that cannot be subsequently changed, and likewise a meaningful alias:

To assign services to service groups you need the rule set Assignment of services to service groups. The fastest way to go there is via the overview of the service groups (Setup > Services > Service Groups). Here you simply click Service Groups > Assign to group > Rules in the menu bar. Alternatively, you can, of course, find the rule via the rule search in the Setup menu, or dig through Setup > Services > Service monitoring rules > Various > Assignment of services to service groups. Now use Create rule in folder to do just that. First you specify which service group to assign services to, for example myservicegroup or its alias My service group 1.

The exciting part now follows in the Conditions section. On the one hand, you can use folders, host tags, and explicit host names to make restrictions outside of the services. Secondly, you name the services you would like to group, such as Filesystem and myservice to create a set of file systems. The specification of the services takes place here in the form of regular expressions. This allows you to define groups exactly.

You can check the assignment of services on the detail page of a particular service. Below, by default, is the Service groups the service is member of line.

As already mentioned, the service groups are used in several places: views, NagVis maps, notifications and alert handlers. For new views it is important that you use the Service groups as the data source. Of course, the Views widget also contains predefined views for service groups, for example a clear summary:

With a click on the service group names you will receive a complete view of all of the services of the respective group.

If you use service groups in NagVis maps, you will receive a summary of service groups opened in a menu by hovering over a single icon:

When you use service groups in notifications and alert handlers, they are available as conditions/filters, of which you can use one or more:

Check plug-ins are required to generate services in Checkmk. Each service uses a check plug-in to determine its status, create/maintain metrics, etc. When doing so such a plug-in can create one or more services per host. So that multiple services from the same plug-in can be distinguished, an Item is needed. For example, for the service Filesystem /var the Item is the text /var. In the case of plug-ins that can only generate a maximum of one service per host, CPU utilization) for example, the Item is empty and not shown.

A list of all available check plug-ins can be found under Setup > Services > Catalog of check plugins. Here the individual plug-ins can be searched for, filtered in various categories:

For each plug-in three columns of information will be shown: a description of the service (Type of Check), the name of the check plug-in (Plug-in Name) and its compatible data sources (Agents):