Monitoring log files

The installation package for the Windows agent from Checkmk Community does not contain a Python environment. However, a corresponding cabinet file is already available on your Checkmk server. You can find this file called python-3.cab in the directory ~/share/check_mk/agents/windows or in Checkmk via Setup > Agents > Windows > Windows agent. Copy this file to your Windows host in the directory C:\Program Files (x86)\checkmk\service\install. There is already a file with this name and a file size of 0 byte. You must overwrite this file with the version from the Checkmk server. Then restart the Checkmk agent service. In Windows PowerShell with administrator rights, you can do this with the following command:

Once the Windows service has been restarted, the virtual Python environment will have been automatically installed.

Alternatively, you can also download and install a current Python package from python.org. Make sure to activate the following options during installation:

If you want to start testing immediately after installing Python, it is essential to restart the checkmkservice either via the Windows Task Manager or with the commands specified above, otherwise the service will not know about the new environment variables.

In the commercial editions, first call up the rule for the agent plug-in Setup > Agents > Windows, Linux, Solaris, AIX > Agent rules > Text log files (Windows, Linux, Solaris, AIX). The default setting Deploy the Logwatch plug-in and its configuration should normally be left as it is. However, if you want or need to transfer the configuration file logwatch.cfg to the host in a different way, the Deploy the Logwatch plug-in without configuration option is still available here.

Continue with the option Retention period. The default setting here is one minute, which also corresponds to the preset check interval in Checkmk. This value should always be at least equal to the check interval. This option is primarily responsible for ensuring that no log messages are lost due to a service detection or the manual execution of cmk -d myhost. Further details can be found in the inline help for the option and in the Werk #14451 with which this option was introduced.

Now comes the section of the rule where things really get going—Configure a log file section. We will start directly with the biggest stumbling block of recent years. In the Patterns for log files to monitor (allowing * and ?) field, you will need to specify the log files that you want to monitor. You can do this individually and explicitly or with so-called glob patterns (glob for short). We are using the Python module glob here, for which there is a detailed documentation on docs.python.org. However, we would like to provide you with a few helpful examples right here.

For example, if you enter /var/log/my.log here, Logwatch will monitor just this single log file. If you instead enter /var/log/*log here, Logwatch will monitor all files that end with the character string log and which are located directly in the /var/log directory. If you want to monitor log files in all direct subdirectories of /var/, you can do this with the following glob, for example: /var/*/*log. We explicitly do not offer the glob ** for recursively searching a directory structure here, because we would end up with far too large a hit list far too quickly and leave the actual purpose of Logwatch behind us.

The following table gives you a few more helpful examples of how you can use globs to actually monitor the files that require monitoring without having to specify them all individually:

So when it comes to which files are 'matched' in the first step, we use no regular expressions and this may be enough for you to reach all the files you want.

However, if you now need to filter further, you can use the Regular expression for log file filtering option to apply regular expressions to the hits from step 1 in a second step.

If you have collected all files /var/log/ and its direct subdirectories in the first step with /var/log/ and /var/log/*/*, you could use the regular expression error.log$|err$ to reduce the hit list to all files that end with err.log or err. Caution: The 'dot' (.) is again now an arbitrary character. This could, for example, leave the files /var/log/apache2/error.log, /var/log/mail.err and /var/log/cups/error_log.

As you can see, we have already provided you with two good and powerful tools for selecting the files to be monitored, so that Logwatch can also check the other parameters and contents very quickly in the next step using a manageable file list. You can deepen your knowledge of the latter in the Regular expressions in Checkmk article.

With the Restrict the length of the lines option you can instruct Logwatch to truncate excessively long lines after the specified number of characters.

The following option Watch the total size of the log file is useful for recognizing a defective log rotation. If you set 100 MiB here, you will receive a warning each time a particular log file has exceeded the specified size again.

The maximum number of lines that Logwatch checks per run and file can be restricted with Restrict number of processed messages per cycle and with Restrict runtime of log file parsing you can ensure that Logwatch does not spend too long on a single file that may have been flooded with thousands and thousands of new entries since the last check.

If you activate one of the latter two options, you must also specify what should happen if the specified limit is exceeded. With our default setting, the associated service becomes critical and you receive a message that lines have been skipped or that the maximum runtime has been exceeded.

Handling of context messages is an option with which the volume of transferred data can become very large very quickly. So think carefully about whether only the log message that you think should generate a CRIT or WARN is important to you, or whether all lines that have been added since the last run of Logwatch should be transferred to the Checkmk server. For small log files that only grow by a few lines every minute, the setting Do transfer context is certainly unproblematic. However, if 50 log files are monitored on a host, which suddenly contain 100,000 new lines with a length of 500 characters, we are already in the gigabyte range. In such an event, it may be enough to see that a large number of new messages have been added since the last check in order to initiate a check directly on the host concerned.

If you do need the context—​i.e. the lines before and after the log message that is important to you—​you can limit this to a certain number of lines before and after with the option Limit the amount of context data sent to the monitoring server.

With Limit the amount of data sent to the monitoring server you can limit the size of the transferred data in general.

Process new log files from the beginning is switched off by default. This sometimes leads to astonishment, because Logwatch does not 'recognize' problems that are in log files and pass these on to the Checkmk server. In our opinion, nothing is older than yesterday’s newspaper and so are the log messages that were already in a log file before the first run of Logwatch. During this very first run, Logwatch does nothing more than note how many lines are already contained in the respective log. Only during the second run are the files checked for their content—​i.e. the newly added lines.

Logwatch relies on actually being able to read the log files. Under the hood, Logwatch goes to great lengths to recognize the coding of each log file. However, character encodings that are too exotic can lead to problems. If you can specify the character encoding of the monitored log files, UTF-8 is a very good choice. Otherwise, you can use the option Character encoding that should be used to decode the matching files to explicitly inform Logwatch about the encoding that is expected in the given log files. The output of Logwatch is always encoded in UTF-8—​if a conversion is necessary, Logwatch will perform it with reference to the encoding you specified.

With Duplicated messages management, if you activate this option, you can again save a bit of bandwidth, and the subsequent output in Checkmk will also be more readable. If you activate Filter out consecutive duplicated messages in the agent output, Logwatch counts how often a line was repeated and writes this accordingly in the output instead of repeating the lines.

Finally, the lines in the log files that are of interest to you are now described using one or more regular expressions, and assigned a state. Each line of a log file that is being evaluated will be checked against each of the given regular expressions in order from top to bottom until a match is found. When a match is found, the corresponding state is assigned and the evaluation continues with the next line.

If you want every line containing the word panic to lead to a CRIT in Checkmk, it is sufficient to enter panic in the Pattern(Regex) field after clicking on Add message pattern below Regular expressions for message classification. The CRIT status assignment is preselected when creating a new entry. Additional options are available here to assign a different status to the lines that match the regular expressions you specified:

In addition to the states that can be triggered for individual lines, there are also options for rewriting or summarizing lines. These options also appear in the State drop-down menu.

If necessary, you can also chain multiple entries of type APPEND or REWRITE together. The lines that have been combined or rewritten are not subsequently processed separately by the regular expressions. So, after merging and/or rewriting, processing continues with the next line that does not match the most recently matched pattern.

The option to forward lines with the OK status to the Checkmk server for further processing brings us to an important point that demonstrates just how flexible Logwatch can be—when used correctly.

All of the options explained in this section become entries in the configuration file already mentioned, which is stored on the respective host. If you now want to make changes to the classification of certain messages, you may first have to edit the rule, then bake the agent and install it.

Alternatively, you can first transfer all of the interesting messages to the Checkmk server (for example with the OK state), and then on the Checkmk server (re-)classify them with the Log file patterns rule. In this way you can save yourself the trouble of baking and rolling out the new agent, and after customizing the above-mentioned rule accordingly, will need to—​onetime only—​quickly activate the changes. You can find out exactly how to do this below in the chapter Reclassifying with log file patterns.

In the Checkmk Community you configure the agent plug-in as usual via a text file. As a rule, Logwatch searches for a file called logwatch.cfg in the directories /etc/check_mk (Linux) or c:\ProgramData\checkmk\agent\config\ (Windows). An (almost) minimal configuration could look like this:

First, always enter a glob pattern here, followed by all the options to be applied. This is followed—​with an indentation of one space—​by a letter representing the desired state or function, and finally a regular expression that is compared with each line of the log file. With the above configuration, all new lines added to the /var/log/my.log file since Logwatch’s last run would be checked against the two patterns, a critical message and something that should only trigger a warning. If a line matches one of the regular expressions, the entire line—along with the defined state (e.g., CRIT for lines beginning with C and WARN for lines beginning with W)—is passed to the Checkmk server.

As a site user, you can find a comprehensive example configuration in the file ~/share/check_mk/agents/cfg_examples/logwatch.cfg. This file also contains comments explaining the syntax and meaning of the various letter abbreviations.

As all of the options that you can specify in such a configuration file have already been explained in the section Configuration via the Agent Bakery, only a list and brief description follows here. Refer to the above section for a detailed explanation.