Checkmk Synthetic Monitoring with Robotmk

Multiple components work together to create this end-to-end monitoring, so here is a brief overview.

You can find the Robotmk scheduler under Setup > Agent rules > Robotmk scheduler (Windows). As the rule is quite extensive, here is a look at the empty configuration:

First, the scheduler requires the base directory in which all your test suites are located. Enter this arbitrary, explicit file path under Base directory of suites, for example C:\robots.

The Parallel plan groups that are shown now are a Checkmk-specific concept.

To explain this, we must first go down one hierarchy level: Here you can see the item Sequential plans. Such a sequential plan defines which suites are to be executed with which parameters. Robot Framework will process these suites one after the other. The reason for this is simple: in practice, tests are sometimes run on the desktop and several test suites could get in each other’s way at the same time (think of them stealing each others control of the mouse cursor).

The plan groups are now an encapsulation for sequentially executed plans — and are themselves executed in parallel. Again, the reasoning is simple: this allows test suites that do not rely on the desktop to be executed in their own plans without delay — the test suite used in this article is a good example of such processing.

Back to the dialog: The only explicit setting is the execution interval, which you set under Group execution interval.

Now it’s time to configure the first plan. You can enter any name under Application name. This name does not have to be unique! The name of the application to be monitored makes sense here, for example OnlineShop, or here in this example simply MyApplication. Of course, it can happen that this online store is tested several times, either by other test suites or by the same test suite with different parameters. In such cases, the Variant field is used to achieve unambiguous results despite identical names. For example, if the application OnlineShop is tested once in German and once in English (via corresponding parameters), you could use corresponding abbreviations here. The monitoring will then return results for My OnlineShop_en and My OnlineShop_de.

However, the specification under Relative path to test suite file or folder is necessary. The path is relative to the base directory specified above, e.g. mybot\test.robot for C:\robots\. Alternatively, a directory (with several robot files) can also be specified here, in which case it would simply be mybot.

Continue with the Execution configuration. Under Limit per attempt you define the maximum elapsed time — per attempt — that a test suite may run. With Robot Framework re-executions you can now instruct Robot Framework to repeat test suites completely or incrementally if tests fail. If the individual tests in a test suite are independent of each other, the incremental strategy is the best way to save time. If, on the other hand, the test suite tests a logical sequence, such as "Login → Call up product page → Product in shopping cart → Checkout", the test suite must of course be completely reprocessed. In the end, there is always only one result.

In the case of complete retries, only the results from self-contained suites are taken into account for the final result: If a test fails on its final retry, the test suite is counted as a failure. In the case of incremental retries, the final result is made up of the best partial results: If some tests only run successfully on the third attempt, the final result is also counted as a success. Reminder: The combination of attempts and maximum run times of all plans in a plan group determines their minimum execution interval.

By default, execution via RCC is activated under Automated environment setup (via RCC), for which you must enter two values. Firstly, RCC requires the specification of where the robot.yaml file is located. Its primary purpose is to reference the conda.yaml file, which is responsible for setting up the Python environment, i.e. installing Python and dependencies. This specification is relative to the base directory that you have set above under Relative path to test suite file or folder. The YAML files can be saved in subdirectories, but best practice is the top suite directory. For the above base directory C:\robot\ and the suite directory C:\robot\mybot it is accordingly mybot\robot.yaml.

With the following time limit for building the Python environment, you should bear in mind that sometimes large volumes of data need to be downloaded and set up. Especially for the required browsers, several hundred megabytes are quickly accumulated — but only for the first run. RCC only rebuilds environments if the content of conda.yaml has changed.

Under Robot Framework parameters you have the possibility to use some of the command line parameters of Robot Framework (which are also displayed by the command robot --help). If you want to use additional parameters, the option Argument files will help. A file specified here can contain any robot parameters. Further information about the individual parameters can be found in the inline help.

For our example project, only the option Variables is activated and a variable MYVAR with the value My Value is set. Remember the command Log ${MYVAR} at the top of the file tests.robot? This is the corresponding reference.

At the end of the suite configuration, there are three largely self-explanatory options. Execute plan as a specific user allows Robotmk to be executed in the context of a specific user account. Background: By default, Robotmk is executed in the context of the Checkmk agent (LocalSystem account), which has no authorization to access the desktop. Here a user can be specified who must be permanently logged in to a desktop session and who has access to graphical desktop applications accordingly.

With Assign plan/test result to piggyback host the results of the plan/test can be assigned to another host. For example, if Robot Framework is testing the ordering process of an online store, the results can be assigned to the corresponding web server.

Each test run produces data that is stored under C:\ProgramData\checkmk\agent\robotmk_output\working\suites\. By default, results from the last 14 days are retained, but you should bear in mind that large volumes of data can quickly accumulate here. At least 500 kilobytes of data are generated per run — with more complex test suites and embedded screenshots, for example, this can quickly add up to several megabytes of data. Depending on the execution interval, the size of the report and your documentation requirements, you should intervene in such a situation.

Once here, you can now create further plans in this plan group or further plan groups.

At the end there are two more options, which in turn relate to the complete Robotmk scheduler configuration.

RCC profile configuration allows you to specify proxy servers and hosts to be excluded.

Grace period before scheduler starts can also be very useful: The scheduler starts together with the Checkmk agent before the desktop logon — which, of course, means that any tests on the desktop must fail. The start can be manually delayed using a grace period.

This completes the configuration and you can bake a new agent with the plug-in and then roll it out, manually or via the automatic agent updates.

In monitoring, you will find services for the status of the Robotmk scheduler as well as the individual plans and tests — even if you have not created any separate service rules.

If the scheduler does not receive any data, building the environment probably did not work. A common reason for this are network problems, for example, due to which certain dependencies cannot be loaded. In this case, take a look at the corresponding log file under C:\ProgramData\checkmk\agent\robotmk_output\working\environment_building.

This is a particularly interesting error that you might encounter on fresh Windows systems. The conda.yaml can also contain instructions that are to be executed after the installation of the dependencies — for example, the initialization of the Robot Framework browser. Python commands should therefore be executed here. By default, Windows 11 has aliases for python.exe and python3.exe that refer to the Microsoft Store. You must deactivate these aliases under 'Settings/Aliases for app execution'.