Checkmk Synthetic Monitoring with Robotmk

Multiple components work together to create this end-to-end monitoring, so here is a brief overview.

You can also run test suites directly with CSM without Checkmk. However, the tool must first be set up for this purpose — specifically for the shell you are using. You can specify your shell manually or let CSM detect it itself:

CSM provides detailed instructions on how to integrate it into your shell — here using Powershell as an example:

CSM is essentially a wrapper for Micromamba. The Init command is used to configure the relevant settings and write them directly to the shell configuration so that the settings remain in place throughout the session. If you want to see which code is generated and written to the shell configuration, simply execute the CSM command solo for your shell:

CSM is now ready for use, and you can start working with virtual environments. In the folder containing your CSM automation package, execute the following command to create the environment:

A useful first command would be to list the available virtual environments:

Now it is time to activate the environment you have created:

Once activated, you can execute Robot Framework commands directly in the virtual environment (robot --version).

But of course, this can also be done without manual activation:

In practice, the Robotmk scheduler operates directly with Micromamba on the test host without CSM.

You can find the Robotmk scheduler under Setup > Agent rules > Robotmk scheduler (Windows). As the rule is quite extensive, here is a look at the empty configuration:

First, the scheduler requires the base directory in which all your test suites are located. Enter this arbitrary, explicit file path under Base directory of suites, for example C:\robots.

The Parallel running sequences of plans that are shown now are a Checkmk-specific concept.

To explain this, we must first go down one hierarchy level: Here you can see the item Sequence of plans. Such a sequential plan defines which suites are to be executed with which parameters. Robot Framework will process these suites one after the other. The reason for this is simple: in practice, tests are sometimes run on the desktop and several test suites could get in each other’s way at the same time (think of them stealing each other’s control of the mouse cursor).

The Parallel running sequences of plans are now an encapsulation for multiple of such sequences — and are themselves executed in parallel. Again, the reasoning is simple: this allows test suites that do not rely on the desktop to be executed in their own sequences without delay — the test suite used in this article is a good example of such processing.

Back to the dialog: The only explicit setting is the execution interval, which you set under sequence execution interval.

Now it’s time to configure the first plan. You can enter any name under Application name. This name does not have to be unique! The name of the application to be monitored makes sense here, for example OnlineShop, or here in this example simply MyApplication. Of course, it can happen that this online store is tested several times, either by other test suites or by the same test suite with different parameters. In such cases, the Variant field is used to achieve unambiguous results despite identical names. For example, if the application OnlineShop is tested once in German and once in English (via corresponding parameters), you could use corresponding abbreviations here. The monitoring will then return results for My OnlineShop_en and My OnlineShop_de.

However, the specification under Relative path to test suite file or folder is necessary. The path is relative to the base directory specified above, e.g. mybot\test.robot for C:\robots\. Alternatively, a directory (with several robot files) can also be specified here, in which case it would simply be mybot.

Continue with the Execution configuration. Under Limit per attempt you define the maximum elapsed time — per attempt — that a test suite may run. With Robot Framework re-executions you can now instruct Robot Framework to repeat test suites completely or incrementally if tests fail. If the individual tests in a test suite are independent of each other, the incremental strategy is the best way to save time. If, on the other hand, the test suite tests a logical sequence, such as "Login > Call up product page > Product in shopping cart > Checkout", the test suite must of course be completely reprocessed. In the end, there is always only one result.

In the case of complete retries, only the results from self-contained suites are taken into account for the final result: If a test fails on its final retry, the test suite is counted as a failure. In the case of incremental retries, the final result is made up of the best partial results: If some tests only run successfully on the third attempt, the final result is also counted as a success. Reminder: The combination of attempts and maximum run times of all plans in a sequence determines their minimum execution interval.

By default, execution via Conda is activated under Automated environment setup, for which you must enter three values. Firstly, Conda requires the specification of where the robotmk-env.yaml file is located. This file is responsible for setting up the Python environment, i.e. installing Python and dependencies. This specification is relative to the base directory that you have set above under Relative path to test suite file or folder. The YAML files can be saved in subdirectories, but best practice is the top suite directory. For the above base directory C:\robot\ and the suite directory C:\robot\mybot1 it is accordingly mybot1\robotmk-env.yaml.

This is followed by the path specification for robotmk-setup.yaml, which contains instructions to be executed after the Conda environment has been built — but only if you are using this file, of course.

With regard to the following time limit Environment build timeout for building the Python environment, please note that large volumes of data may need to be downloaded and set up. The required browsers in particular can quickly add up to several hundred megabytes — but only during the initial run.

Under Robot Framework parameters you have the possibility to use some of the command line parameters of Robot Framework (which are also displayed by the command robot --help). If you want to use additional parameters, the option Argument files will help. A file specified here can contain any robot parameters. Further information about the individual parameters can be found in the inline help.

For our example project, only the option Variables is activated and a variable MYVAR with the value My Value is set. Remember the command Log ${MYVAR} at the top of the file sample.robot? This is the corresponding reference.

The Secret environment variables option deserves special attention as it is not an original Robot Framework function. You can set secret environment variables here, intended for example for passwords in conjunction with the CryptoLibrary Robot Framework library. The variables set here do not appear in logs, but are written in plain text to the Checkmk configuration files on the respective test hosts.

At the end of the suite configuration, there are three largely self-explanatory options.

The first option exists exclusively for Windows hosts. Execute plan as a specific user allows Robotmk to be executed in the context of a specific user account. Background: By default, Robotmk is executed in the context of the Checkmk agent (LocalSystem account), which has no authorization to access the desktop. Here a user can be specified who must be permanently logged in to a desktop session and who has access to graphical desktop applications accordingly.

However, do not be too hasty: browsers are an exception here! They can run and render web pages in headless mode. You should only check the box for a specific user if a dedicated desktop application is launched outside of browsers — not least for security reasons.

With Assign plan/test result to piggyback host the results of the plan/test can be assigned to another host. For example, if Robot Framework is testing the ordering process of an online store, the results can be assigned to the corresponding web server.

Each test run produces data that is stored under C:\ProgramData\checkmk\agent\robotmk_output\working\suites\. By default, results from the last 14 days are retained, but you should bear in mind that large volumes of data can quickly accumulate here. At least 500 kilobytes of data are generated per run — with more complex test suites and embedded screenshots, for example, this can quickly add up to several megabytes of data. Depending on the execution interval, the size of the report and your documentation requirements, you should intervene in such a situation.

Once here, you can now create further plans in this sequence or further sequences.

At the end there are two more options, which in turn relate to the complete Robotmk scheduler configuration.

Conda configuration allows you to specify proxy servers, TLS and SSL certificates, as well as hosts to be excluded.

Grace period before scheduler starts can also be very useful: The scheduler starts when the test host is restarted together with the Checkmk agent before the desktop logon — which, of course, means that any tests on the desktop must fail. A grace period can be used to delay the start manually so that, for example, the technical user’s desktop becomes active (again) via Autologin (Windows function for logging in without manual password entry).

This completes the configuration and you can bake a new agent with the plug-in, and then deploy, manually or via the automatic agent updates.

In monitoring, you will find services for the status of the Robotmk scheduler as well as the individual plans and tests — even if you have not created any separate service rules.

In principle, you could simply archive the complete robot directory with the standard files (robotmk-env.yaml, robotkmk-setup.yaml, sample.robot etc.).

However, such tests are often managed via Git and consequently there are regularly files that should not be distributed — these are typically managed via a gitignore file. As an aid to ensuring clean archives, here are two small scripts for Windows and Linux that create archives without the files to be ignored. Use these scripts only as an aid and always test the functionality for your system beforehand.

For this variant, you do not need to make any changes to your tests themselves. Simply open the Robotmk KPI discovery rule and enter the keywords to be monitored as regular expressions or very specific names.

Caution: If the regular expression matches more than one keyword in the same test, only one keyword will be recognized and its service status will be UNKNOWN (because Checkmk does not know which match is actually intended).

For a discovery via marker, you must perform the following steps:

For the installation, the robotmk-env.yaml from above must be extended with robotframework-robotmklibrary. Modifications compared to the suite above are highlighted in yellow:

The test suite sample.robot must be extended by the library (in the Settings area) and a marker. KPIs will often refer to individual user keywords, so the keyword Foobar is added here (which only calls the standard keyword Log and is in turn called by the test case Test1).

The Robotmk keyword Monitor Subsequent Keyword Runtime is the marker to (trigger Checkmk to) monitor the following keyword (Foobar); more precisely, its runtime. The optional discover_as argument can be used to specify an individual name for the service in the monitoring — the keyword Foobar therefore appears in the monitoring as a service called My user keyword.

The big advantage here compared to pattern-based discovery is that the same keyword can also be explicitly monitored multiple times within a single test.

Here is the example above, extended by two Foobar calls:

The three calls to the Foobar keyword would therefore appear in the monitoring as My user keyword, Foobar_2 and Foobar_3.

Regardless of which of the two variants the keywords are used for monitoring, the next step is always to configure the evaluation: At what runtime should the services go to WARN or CRIT? To do this, define the corresponding levels in the Robotmk KPI monitoring rule.

The keyword services appear in monitoring under one of these two patterns:

The difference is therefore only in the indication of origin in brackets directly before the keyword.

As shown in the above screenshot with two Foobar keyword services, the suite needed to be extended somewhat:

The two keywords Foobar and Barfoo both appear under the name Foobar in the monitoring, but can be distinguished by the indication of origin in brackets.

The example with two different keywords that are listed under the same name primarily serves to clarify the origin information.

As already mentioned above, the actual purpose of discover_as is to be able to call up one keyword multiple times in a test but to name each occurrence individually.

The easiest way to monitor events in Synthetic Monitoring is to track the log files in a live view. The exact storage paths can be found in the overview below. Under Linux the following command can be used for this purpose:

On Windows, you could use Notepad++ or VS Code, for example, but many other IDEs also offer the option for live tracking of file changes.

Depending on the configuration of your plans, tests, and environments, this process may take some time, as it is primarily used to track regular scheduler activity.

Alternatively, the scheduler can also be started manually. To do this, the scheduler must first be stopped and an maintenance time should be set in the monitoring to prevent any impact on the monitoring and to prevent the scheduler from restarting automatically.

In Windows, you can stop the service using the Task Manager. In Linux, you can stop and start the service in the terminal:

Starting the scheduler manually gives you a little more control, but you will still have to rely on log files, and the complete synthetic monitoring configuration will be executed.

However, the scheduler also supports manual and one-time execution of individual plans. In this case, execution takes place immediately; any waiting times or other execution logic are irrelevant here. In addition, you receive the information directly in the terminal without having to go through log files. Even for ad hoc execution, you must first stop the currently running scheduler service and set a maintenance time in the monitoring. In this scenario, the scheduler also loads the complete configuration, but restricts execution to the specified plan:

The plan ID specified here corresponds to the entry you made in the Robotmk scheduler configuration under Sequence of plans > Application name, followed by an underscore and the actual bot name, in this case, for example, myapp1_mybot1 (as the service also appears in monitoring).

The -vv argument for extended output is also important, as otherwise the scheduler provides little feedback.

A final available argument, --no-plan-result, prevents a result, i.e., a Robot Framework report, from being copied to the corresponding folder, so that it remains only locally.

If the scheduler does not receive any data, building the environment probably did not work. A common reason for this are network problems, for example, due to which certain dependencies cannot be loaded. In this case, take a look at the corresponding log file under C:\ProgramData\checkmk\agent\robotmk_output\working\environment_building, respectively /var/lib/check_mk_agent/robotmk/scheduler/environment_building..

Attention: Under Linux, the Robotmk scheduler does not create its own log file (under Windows the robotmk_scheduler_rCURRENT.log), but instead logs via agent and syslog. The corresponding command: