When setting up the monitoring, certainly the most important task is the administration of the systems to be monitored — the Hosts. It’s not just about registering the correct master data (e.g., host name, IP address) — settings for the monitoring (e.g., alarms, threshholds, etc.) also need to be attended to.
Checkmk has been developed from its beginning for environments with a large number of hosts. In order for the configuration to be manageable for the user, Checkmk pursues a different approach to configuration than other systems that originated from the Nagios ecosystem. The most important principles are:
1.1. Folders and their hierarchies
Everyone who works with computers knows the principles of data sets and folders. WATO uses a similar principle for administering hosts, which in effect take on the role of data sets. Insofar as folders themselves can be in folders, the result is a ‘tree structure’. There are three widely-used criteria for building the host-tree:
Location (e.g. Munich versus Shanghai)
Host type (e.g. Switch versus Loadbalancer)
Organisation structure (e.g. Database group versus Networker)
Naturally you can also mix these criteria in a tree with, for example, subdivision by location in the first level, and by host type in the second.
If you love simple things you should pack the actual hosts only in the tree’s ‘leaves’ (although Checkmk also allows hosts in intermediate folders). The following example shows a simple tree structured by host type: The hosts A, B and C are in the folder Servers and D, E and F in Network :
1.2. Attribute inheritance
If you build the tree cleverly you can use it to pass on attributes in a meaningful manner. This is especially useful with attributes that are the same for large groups of hosts, e.g., the SNMP community, or Host characteristics such as Agent type, with which you define whether the host should be monitored per SNMP or via a Checkmk agent.
The following example shows the passing-on of the Agent Type attribute
snmp-only values, likewise the
Criticality attribute with the
Attributes defined lower in the tree always have precedence. Values defined
directly at the host therefore overwrite everything that comes from the
folders. In the above example, the host A receives the
cmk-agent attributes, host D receives
snmp-only, and host F — because of the explicit attribute
test at the host — receives the
A big advantage of this procedure over the the widely-used copy & paste approach of data base oriented configuration systems is that you can PREdefine attributes for hosts that will be registered in the future. This makes your (or your colleagues’) work easier — simply throw the new host into the correct folder and all settings will automatically be correct!
A further function of the folder is the assignment of permissions for creating and editing hosts. Checkmk here differentiates between rights in WATO and the contact allocation in monitoring. It’s not always the case that the persons authorised to create a host are the same people who are responsible for the host’s operational monitoring. The permissions are explained in their own article.
You create new folders via the button. The options are the same as when creating new hosts, which we will explain in detail below.
2. Creating hosts in WATO
You can manage folders and hosts via the Hosts WATO module:
The create host button, the clone button and the edit host button take you to the page with the host’s attributes. This consists of three sections:
2.1. The host name
Most important is the host name. Everywhere in Checkmk this field serves to explicitely identify the host. The host name is entered in internal references, used as a component of the URL, serves as a part of file names and indexes, and appears in log files, etc. There is in fact a function for changing host names at a later date — this is however a time-consuming and complex procedure that is best avoided. You should therefore select host names carefully. The host’s name does not necessarily need to match the host’s DNS name, but it makes many things easier.
2.2. Alias and IP addresses
You can give the host an alternative, descriptive name which will be displayed in many locations in the GUI as well as in reports. If no alias is defined, the host’s name will be used as an alias.
You have four options for configuring the IP address:
You enter no IP address. The host name must be resolvable via DNS.
with Activate changes
You enter an IP address – in the standard format.
Instead of an IP address you can alternatively enter a DNS-resolvable host name.
during check execution
Via rules set Hosts with dynamic DNS lookup during monitoring you determine hosts for a dynamic DNS. The result is similar to 3, except that the host name field is used for DNS query.
during check execution
With the host name method Checkmk uses cached data in order to minimise repeated DNS requests during an Activate Changes — which is very important for accelerating the activation procedure. Furthermore, the cache ensures that a changed configuration can still be activated if the DNS stops working.
The catch is that Checkmk doesn’t automatically notice the change to
an address in DNS. For this reason, in the host details there is the
button which deletes the entire DNS cache and
forces a new resolution at the next Activate changes. This file is found
~/var/check_mk/ipaddresses.cache in your instance, by the way.
Deleting this file has the same effect as the button as described above.
Checkmk incidentally also supports monitoring via IPv6 — also in dual stack.
2.3. Data Sources and Custom Attributes
The final important setting can be performed in the Data Sources and Custom Attributes boxes. The attributes shown here can be extended as desired, and can be used via rules to configure all host and service parameters very efficiently.
It is more important to specify a data source under Data Sources, since this determines how the host transfers its data to the monitoring.
As data sources you can set:
Normal Checkmk agent
The host should be monitored via the Checkmk agents (which must be installed of course). Select this setting also in the case of special agents, such as e.g., ESX-Monitoring
The host should be monitored via SNMP. This selection allows the SNMP Community field to appear in Basic settings, with which you can define the SNMP-Community. Since this is generally the same for many hosts, it is rather recommended that it be defined in a folder. If nothing is specified
Piggyback-Data from other hosts will be used.
Such hosts are without agents and are monitored only with Active checks. Rules for these are found under Host & Service Parameters > Active checks in WATO. If you don’t define at least one active check then Checkmk creates a PING service automatically.
You can find the No agent setting as an option of Check_MK Agent.
Starting with version 1.6.0 of Checkmk there is the new concept of Labels. A host can have any number of labels. Labels are similar to host tags, but unlike those, they do not have to be predefined — you can assign them freely.
Enter labels for the hosts by clicking Add some Label with the mouse. Press Enter after each label to complete it! Use the crosses to remove labels.
If you don’t really need a value for a label, but just want to know whether a
certain label is attached to the host or not, you can simply enter
as value (
vm:yes). If you follow this scheme consistently, you will
find it easier to define conditions for such labels later.
By the way, labels can also be automatically attached to hosts: on the one hand by external connectors that automatically create hosts (e.g. automatically detected hosts in cloud environments), on the other hand by rules.
2.5. Saving and more
After creating or cloning a host the next logical step is always Save & go to Services. With this you enter the automatic service detection, a subject we want to address in the next section. Save & Test takes you into the diagnosis mode – with which you can test whether the settings being used produce ANY data at all from the agent. Details about the diagnosis mode can be found in the article on the agents.
3. Configuring services
After creating a host the next step is the configuration of its services to be monitored. All details for the automatic detection and configuration of the services can be found in its own article. We will describe only the most important here.
There are various ways of accessing the list of a host’s configured services in WATO:
with the Save & go to Services button on a host’s detail page
with the button on a host’s detail page (without saving)
with the symbol on the list of hosts in a folder
in the menu, by selecting the Check_MK Discovery service with the Edit Services entry
A few relevant tips:
The usual method when creating a new host is to use the Save manual check configuration button, which adopts all services to be found for monitoring (Available (missing) services).
If you open an existing host’s page and find services that are not currently being monitored, then the Activate missing button is a sensible tool — this adds the missing services.
The Full scan button enables fresh, complete data to be obtained from a target device. Checkmk works with cached data to enable the rapid loading of pages for a normal monitoring’s displays. With SNMP devices the button starts an active search for new check plug-ins and can possibly find further services.
Automatic Refresh is the same as a clearing and fresh detection of all services. This is useful for services which can recall the state detected by a discovery (e.g., the current state of switch ports).
Via the check boxes you can select or deselect individual services. This is only a temporary solution as the service detection always highlights missing services. To permanently ignore a service requires the creation of a rule, and is achieved with the symbol.
As always after every change an Activate Changes is necessary in order for them to take effect.
All further information can be found in the article on Service configuration.
4. Bulk operations
You may occasionally wish to perform tasks such as deleting, moving, editing or service detection for a whole series of hosts simultaneously. WATO provides so-called bulk operations for this purpose. These always apply for hosts that are located directly in a folder. You can restrict the selection by entering a search text to the left of Search, or via check boxes which you activate with . With a final click on one of the buttons in the Bulk bar the operation will be carried out or at least be initiated for all hosts.
Here are a few tips for the less self-explanatory operations:
4.1. Edit and cleanup
Edit enables changes to one or more attributes on all selected hosts. The attribute is thereby entered explicitely in the hosts. Attention: there is a difference between the host inheriting an attribute from a folder, and the attribute being set explicitly. Why? In the latter case a change to the attribute in the folder would have no effect, as the values defined directly in the host always have priority.
The Cleanup operation is available for this reason. With this you can delete explicit attributes from the selected hosts and reinstate inheritance. The same result can be achieved by opening every host individually and deselecting the attributes via the check boxes.
It is generally a good idea to use as few explicit attributes as possible. When everything is inherited correctly via the folders, errors are reduced and the easy integration of new hosts is made possible.
You can find details about Discovery in the article on Services.
5. Host searches in WATO
WATO offers its own search function for configured hosts, with which you can search beyond the limits of folders. Why can’t you simply search via the views in monitoring? That would certainly work with the search for a single host. You could access this host via the symbol in WATO.
But let us remind ourselves: in the Introduction to WATO article we saw that the hosts in the configuration environment are not necessarily the same as those in the operational monitoring environment. The WATO search additionally offers the possibility of performing bulk operations immediately on the discovered hosts.
The search can be reached via the button you can find in every folder. The search always preceeds from the current folder recursively through all subfolders. To search globally, simply use the search from the main folder. In the Hostname field an infix search is valid — the entered text must only be a part of the host name. Furthermore, you can restrict the search with characteristics or other attributes:
All search terms are connected with AND. The example in the above image illustrates a search for all hosts with the Test system attribute that also include ora in their name.
The resulting list behaves almost like a normal folder. This means that here you can work with Bulk operations, in order, for example, to move all discovered hosts into a specific folder. If you don’t like the results, you can adjust and refine the seach at any time with .
6. Importing hosts from CSV data
If you wish to import a large number of hosts from a previous monitoring system or from an Excel table, you can make the task easier by importing with the help of CSV data. Checkmk is very flexible when reading such CSV data. In the simplest case you just need a file in which every line contains a host name that can be resolved via DNS:
myserver01 myserver02 myserver03
During an import it is also possible to take on additional attributes. If the CSV data has attribute names in the first line, Checkmk can even assign these automatically. To this end Checkmk attempts to use a tolerant rather than an exact syntax. In the following data WATO can automatically correlate all four columns correctly:
hostname;ip address;alias;agent srvlnx17;10.0.0.10;web99;cmk-agent srvlnx18;10.0.0.32;Backupserver;cmk-agent switch47-11;;Backpserver23;snmp-only
The procedure is as follows: select or create a target folder for the import. Switch to this folder and click on . In the dialogue that opens either upload the data, or select Content of CSV file and copy the content into the input field that opens. You can even automatically perform an immediate service discovery on the newly-imported hosts with the Perform automatic service discovery option:
Selecting a separator in the next step is not necessary here, as it will be recognised automatically. Here you select the Has title line option:
A click on Update preview displays the following table:
If the automatic recognition of a column doesn’t work you can manually-select
the attribute to be assigned. Under the host attributes in the
CSV data it is essential that the attribute’s internal name be used (here
cmk-agent, and not Checkmk agent (server)). The exact
internal names can found with Host Attributes in the
If you have earlier selected Perform automatic service discovery, the same mask as used in Bulk discovery appears. After the discovery completes, all that is needed is the familiar Activate Changes for all of the new hosts to be in the monitoring!
7. Creating parents
7.1. Creating parents manually
You have already learned how Parents functions, and what the states of Hosts and Notifications are all about. But how does one actually create Parents? The answer is typically Checkmk: there are a number of different procedures — manually, per scan, or via the Web-API.
A parent for a single host can be specified as follows: In WATO > Hosts open the desired host’s attributes. In the Basic Settings section enter the parent using its name or IP-address. Once a parent has been specified, a further entry field for an additional parent will be opened.
Important: Only direct Parent-Hosts may be specified.
Similarly, parents can also be defined in a folder’s attributes, and be inherited by the hosts they contain. How this is achieved has already been seen in the section on Host-Management.
7.2. Creating parents using a scan
If the monitoring is a new installion, which from the very beginning has been planned with an orderly folder and parents structure, there should be no real problems with the inheriting of parents via folders. Parents can also be set up automatically using a scan. The Parent Scan can be found in WATO > Hosts in each individual folder.
Via the IP-Protocol the scan searches for the last Gateway before a host on the OSI-Model’s (Layer 3) Network Layer using traceroute. If such a Gateway is found and its address belongs to one of your monitored hosts, this host will then be set as a parent. If the Hop’s traceroute receives no information from the targeted host, the info from the last successful Hop will be used.
If however no gateway with a monitored IP-address is found, as standard Checkmk generates an artificial Ping-only-Host in the Parent folder which will be simultaneously generated by default.
This standard setting can also produce undesirable results. For example, take a typical, small network with the address range 192.168.178.0/24. If a host with an address in a different address space — which cannot be pinged — is added to this monitoring, the scan will attempt to access it via the router, and there it will find only a net-provider node. Thus, for example, it can happen that a telecom-server in the WAN-network is defined as a parent for this host. This option can of course be deactivated.
If you wish to scan a folder with new hosts for parents, proceed as follows:
First navigate to the desired folder and click on the Parent scan icon.
The Scan-Configuration will open. To fully scan in all hosts in all subfolders, independently of possible manually-installed parents, under Selection choose the Include all subfolders and Scan all hosts options. In the Performance menu you can limit the scan-duration – which otherwise can take a very long time if there is a large number of hosts.
In Creation of gateway hosts specify if, how, and under which alias new parent-hosts should be created. Deactivate this function if it is to be restricted to parents on monitored hosts.
Now start the scan. The scan’s output can be followed live. On completion the changes must as usual be activated with Activate Changes. Finally the configured parents and, if applicable, a new folder Parents can be viewed in WATO > Hosts.
With this the scan has been completed.
Following a scan process the Parent-Child relationship will be automatically opened as a topological diagram, which can also be displayed with Views > Network Topology.
Tip: If the result of a scan appears to be implausible at any point, invoking a manual traceroute can sometimes help with analysing the individual hops.
By the way — one can also scan selected hosts, rather than a complete folder: in activate the check boxes, select the desired hosts, and start the group-action Parentscan.
7.3. Creating parents without WATO
For more experienced users there is the additional facility for configuring parents by using Web-API.
8. Renaming hosts
Renaming hosts — on the face of it a simple matter — turns out to be an astoundingly-complex operation on closer inspection. The reason for this is that Checkmk uses the host’s name as the unique key for the host – and this is used in numerous locations. These include log data, file names, configuration rules, BI agreggations, reports, dashboards and much more. The host name also appears in URLs.
WATO has a specific function for cleanly-renaming a host in all locations. In a host’s details you can rename it by using the button, or in a folder rename multiple hosts simultaneously with the button.
By utilising intelligent operations, Bulk Renaming allows systematic
name matching to be made. In the Hostname matching field you optionally
enter a regular expression that matches the first characters of
the names of the hosts that you wish to rename — here as an example, all hosts
whose names begin with
mysrv. Then enter one or more operations
in the sequence that they should be applied to the hosts. In the
following example, for all hosts everything after the first
will be truncated and replaced by the ending
Numerous operations are available. Please activate the Online Help , and select the operation to receive an explanation about it. Following the obligatory “Are you sure…?” query…
… the processing can take a while. During the renaming the monitoring will be *completely stopped*! This is necessary to keep everything in a consistent state. On completion you will receive on overview listing which and where renames have taken place:
9. Host groups
9.1. Why have host groups?
Host groups are a part of the monitoring basics in Checkmk. They enable a second layer of groups of hosts across the folder structure. For example: your locations are displayed based on the folders. Now you would like to be able to view all Linux, or particular application-servers together. By using a host group you can generate suitable views, create NagVis-maps, and likewise customise notifications and alert handlers. In contrast to the situation in host tags, host groups do not appear as selection criteria in rules: host groups serve the views, while host tags serve the configuration. Host groups can be found under WATO > Host & Service Groups:
9.2. Creating and editing host groups
A new host group is created using . The creation is a trivial action, and is limited to specifying a unique name which cannot be changed later, and likewise an alias:
To finish, as usual the modifications must be activated with Activate Changes.
9.3. Including hosts in a host group
To add hosts to a host group, try the Assignment of hosts to host groups rule set, which can be found under WATO > Host & Service Parameters > Grouping. Create a new rule in the desired folder with . Next, in the Assignment of hosts to host groups panel specify to which host group the hosts are to be assigned — in the example below something like the group myhostgroup, or respectively its alias My Host Group:
Finally, in the Conditions panel, attend to these or to the filters. You can filter hosts by host tags and folders, or specify particular hosts. Filters can of course also be combined to restrict the group. Should you wish to add hosts with two tags from the same attributes group to the host group, you will need to create two separate rules. In general, the group assignments are cumulative. Hosts can be in multiple groups and groups can be filled with multiple rules. You can also specify hosts in the form of regular expressions so that all hosts which include backup but not testing in their names can be captured with a single entry.
9.4. Checking a host’s host groups
You can check the result of your mappings on a host’s status page using in the host properties. Below, by default towards the bottom, is the line Host groups the host is member of:
9.5. Using host groups
As mentioned above, you can use host groups in three places: you can create views, build NagVis maps, and they can be used as filters in rules for notifications and alert handlers. Only the specification of Hostgroups as the data source is important. The Views widget of course includes ready-made views, such as this handy summary:
Click on the names of the host groups to get a complete view of the hosts in this group.
When used in NagVis maps, for example, you get summaries of host groups via a hover menu over each icon:
10. The folder structure in the monitoring view
The tree structure derived from the folders is also visible to their users in monitoring. On the one hand, there is a WATO Folder filter in all views that you can use to restrict the current view to only those hosts below a particular folder:
On the other hand, via the Folders sidebar element you can restrict the view on the right side to a single folder:
This element functions in conjunction with the Views element. Once selected, a folder is retained even if you select another view. This works for dashboards as well. Try it for yourself!