Checkmk Synthetic Monitoring with Robotmk

Multiple components work together to create this end-to-end monitoring, so here is a brief overview.

You can find the Robotmk scheduler under Setup > Agent rules > Robotmk scheduler (Windows). As the rule is quite extensive, here is a look at the empty configuration:

First, the scheduler requires the base directory in which all your test suites are located. Enter this arbitrary, explicit file path under Base directory of suites, for example C:\robots.

The Parallel plan groups that are shown now are a Checkmk-specific concept.

To explain this, we must first go down one hierarchy level: Here you can see the item Sequential plans. Such a sequential plan defines which suites are to be executed with which parameters. Robot Framework will process these suites one after the other. The reason for this is simple: in practice, tests are sometimes run on the desktop and several test suites could get in each other’s way at the same time (think of them stealing each others control of the mouse cursor).

The plan groups are now an encapsulation for sequentially executed plans — and are themselves executed in parallel. Again, the reasoning is simple: this allows test suites that do not rely on the desktop to be executed in their own plans without delay — the test suite used in this article is a good example of such processing.

Back to the dialog: The only explicit setting is the execution interval, which you set under Group execution interval.

Now it’s time to configure the first plan. You can enter any name under Application name. This name does not have to be unique! The name of the application to be monitored makes sense here, for example OnlineShop, or here in this example simply MyApplication. Of course, it can happen that this online store is tested several times, either by other test suites or by the same test suite with different parameters. In such cases, the Variant field is used to achieve unambiguous results despite identical names. For example, if the application OnlineShop is tested once in German and once in English (via corresponding parameters), you could use corresponding abbreviations here. The monitoring will then return results for My OnlineShop_en and My OnlineShop_de.

However, the specification under Relative path to test suite file or folder is necessary. The path is relative to the base directory specified above, e.g. mybot\test.robot for C:\robots\. Alternatively, a directory (with several robot files) can also be specified here, in which case it would simply be mybot.

Continue with the Execution configuration. Under Limit per attempt you define the maximum elapsed time — per attempt — that a test suite may run. With Robot Framework re-executions you can now instruct Robot Framework to repeat test suites completely or incrementally if tests fail. If the individual tests in a test suite are independent of each other, the incremental strategy is the best way to save time. If, on the other hand, the test suite tests a logical sequence, such as "Login > Call up product page > Product in shopping cart > Checkout", the test suite must of course be completely reprocessed. In the end, there is always only one result.

In the case of complete retries, only the results from self-contained suites are taken into account for the final result: If a test fails on its final retry, the test suite is counted as a failure. In the case of incremental retries, the final result is made up of the best partial results: If some tests only run successfully on the third attempt, the final result is also counted as a success. Reminder: The combination of attempts and maximum run times of all plans in a plan group determines their minimum execution interval.

By default, execution via RCC is activated under Automated environment setup (via RCC), for which you must enter two values. Firstly, RCC requires the specification of where the robot.yaml file is located. Its primary purpose is to reference the conda.yaml file, which is responsible for setting up the Python environment, i.e. installing Python and dependencies. This specification is relative to the base directory that you have set above under Relative path to test suite file or folder. The YAML files can be saved in subdirectories, but best practice is the top suite directory. For the above base directory C:\robot\ and the suite directory C:\robot\mybot it is accordingly mybot\robot.yaml.

With the following time limit for building the Python environment, you should bear in mind that sometimes large volumes of data need to be downloaded and set up. Especially for the required browsers, several hundred megabytes are quickly accumulated — but only for the first run. RCC only rebuilds environments if the content of conda.yaml has changed.

Under Robot Framework parameters you have the possibility to use some of the command line parameters of Robot Framework (which are also displayed by the command robot --help). If you want to use additional parameters, the option Argument files will help. A file specified here can contain any robot parameters. Further information about the individual parameters can be found in the inline help.

For our example project, only the option Variables is activated and a variable MYVAR with the value My Value is set. Remember the command Log ${MYVAR} at the top of the file tests.robot? This is the corresponding reference.

The Secret environment variables option deserves special attention as it is not an original Robot Framework function. You can set secret environment variables here, intended for passwords in conjunction with the 'CryptoLibrary' Robot Framework library. The variables set here do not appear in logs, but are written in plain text to the Checkmk configuration files on the respective test hosts.

At the end of the suite configuration, there are three largely self-explanatory options.

Execute plan as a specific user allows Robotmk to be executed in the context of a specific user account. Background: By default, Robotmk is executed in the context of the Checkmk agent (LocalSystem account), which has no authorization to access the desktop. Here a user can be specified who must be permanently logged in to a desktop session and who has access to graphical desktop applications accordingly.

With Assign plan/test result to piggyback host the results of the plan/test can be assigned to another host. For example, if Robot Framework is testing the ordering process of an online store, the results can be assigned to the corresponding web server.

Each test run produces data that is stored under C:\ProgramData\checkmk\agent\robotmk_output\working\suites\. By default, results from the last 14 days are retained, but you should bear in mind that large volumes of data can quickly accumulate here. At least 500 kilobytes of data are generated per run — with more complex test suites and embedded screenshots, for example, this can quickly add up to several megabytes of data. Depending on the execution interval, the size of the report and your documentation requirements, you should intervene in such a situation.

Once here, you can now create further plans in this plan group or further plan groups.

At the end there are two more options, which in turn relate to the complete Robotmk scheduler configuration.

RCC profile configuration allows you to specify proxy servers and hosts to be excluded.

Grace period before scheduler starts can also be very useful: The scheduler starts together with the Checkmk agent before the desktop logon — which, of course, means that any tests on the desktop must fail. The start can be manually delayed using a grace period.

This completes the configuration and you can bake a new agent with the plug-in, and then deploy, manually or via the automatic agent updates.

In monitoring, you will find services for the status of the Robotmk scheduler as well as the individual plans and tests — even if you have not created any separate service rules.

In principle, you could simply archive the complete robot directory with the standard files (robot.yaml, conda.yaml, test.robot) — under Windows as a ZIP, under Linux as a tar.gz.

However, such tests are often managed via Git and consequently there are regularly files that should not be distributed — these are typically managed via a gitignore file. As an aid to ensuring clean archives, here are two small scripts for Windows and Linux that create archives without the files to be ignored. Use these scripts only as an aid and always test the functionality for your system beforehand.

Windows:

Linux:

For this variant, you do not need to make any changes to your tests themselves. Simply open the Robotmk KPI discovery rule and enter the keywords to be monitored as regular expressions or very specific names.

Caution: If the regular expression matches more than one keyword in the same test, only one keyword will be recognized and its service status will be UNKNOWN (because Checkmk does not know which match is actually intended).

For a discovery via marker, you must perform the following steps:

For the installation, the conda.yaml from above must be extended with robotframework-robotmklibrary. Modifications compared to the suite above are highlighted in yellow:

The test suite tests.robot must be extended by the library (in the Settings area) and a marker. KPIs will often refer to individual user keywords, so the keyword Foobar is added here (which only calls the standard keyword Log and is in turn called by the test case My Test).

The Robotmk keyword Monitor Subsequent Keyword Runtime is the marker to monitor (to trigger Checkmk) the following keyword (Foobar); more precisely, its runtime. The optional discover_as argument can be used to specify an individual name for the service in the monitoring — the keyword Foobar therefore appears in the monitoring as a service called My user keyword.

The big advantage here compared to pattern-based discovery is that the same keyword can also be explicitly monitored multiple times within a single test.

Here is the example above, extended by two Foobar calls:

The three calls to the Foobar keyword would therefore appear in the monitoring as My user keyword, Foobar_2 and Foobar_3.

Regardless of which of the two variants the keywords are used for monitoring, the next step is always to configure the evaluation: At what runtime should the services go to WARN or CRIT? To do this, define the corresponding levels in the Robotmk KPI monitoring rule.

The keyword services appear in monitoring under one of these two patterns:

The difference is therefore only in the indication of origin in brackets directly before the keyword.

As shown in the above screenshot with two Foobar keyword services, the suite needed to be extended somewhat:

The two keywords Foobar and Barfoo both appear under the name Foobar in the monitoring, but can be distinguished by the indication of origin in brackets.

The example with two different keywords that are listed under the same name primarily serves to clarify the origin information.

As already mentioned above, the actual purpose of discover_as is to be able to call up one keyword multiple times in a test but to name each occurrence individually.

The practice is far simpler than the technical background — especially when working with a catalog as a ZIP archive (or tar.gz archive under Linux). Basically, you just build the catalog by specifying the ROBOCORP_HOME variable, export it as an archive and then distribute it manually. Below is the path in detail. The paths in the following examples are limited to Windows — only if Linux requires different commands will these be specified explicitly.

First of all: RCCRemote is not a stand-alone tool, you will not find any documentation or even a product page from Robocorp/Sema4. The remote server is merely a part of RCC and is built when RCC is compiled. The server is usually operated via a container. For container operation, there is now a project on the GitHub pages of Elabit, the developer of the original Robotmk plug-in, called RCCRemote-Docker. However, RCCRemote and RCCRemote-Docker are not parts of Checkmk Synthetic Monitoring and are not covered by Checkmk support!

Catalogs for this distribution path can be created in two ways: Catalogs exported as ZIP files (see above) can be used for both Windows and Linux. For Linux only, it is also possible to build the catalogs directly in the RCCRemote-Docker container, which completely eliminates the need for manual intervention.

You can find out how to set up RCCRemote-Docker, handle certificates and import catalogs on the project pages.

The configuration on the Checkmk side is now again — typically — quite simple: The Environment dependency handling option is this time set to Fetch from RCC remote server and supplied with the server’s address.

One final setting is still missing, depending on whether your RCCRemote server uses a self-signed or a CA-signed certificate. In both cases, you usually have to set Robotmk scheduler (Windows|Linux) the option RCC profile configuration to Custom profile.

For a self-signed certificate, it is sufficient to leave the Validate TLS certificates box unchecked:

If, however, it is a CA-signed certificate, this option must be activated and the certificate from the signing authority must be stored under Root CA (in PEM format):

If the scheduler does not receive any data, building the environment probably did not work. A common reason for this are network problems, for example, due to which certain dependencies cannot be loaded. In this case, take a look at the corresponding log file under C:\ProgramData\checkmk\agent\robotmk_output\working\environment_building.

This is a particularly interesting error that you might encounter on fresh Windows systems. The conda.yaml can also contain instructions that are to be executed after the installation of the dependencies — for example, the initialization of the Robot Framework browser. Python commands should therefore be executed here. By default, Windows 11 has aliases for python.exe and python3.exe that refer to the Microsoft Store. You must deactivate these aliases under 'Settings/Aliases for app execution'.

Attention: Under Linux, the Robotmk scheduler does not create its own log file (under Windows the robotmk_scheduler_rCURRENT.log), but instead logs via agent and syslog. The corresponding command: