Generating Character-Based Templates for Log Data

Abstract

Log line clusters usually lack meaningful descriptions that are required to understand the information provided by log lines within a cluster. Template generators allow to produce such descriptions in form of patterns that match all log lines within a cluster and therefore describe the common features, e.g., substrings, of the lines. Current approaches only allow the generation of token-based (e.g., space-separated words) templates, which are often inaccurate for log lines, because they usually do not account for existing string similarities in, for instance fully qualified system names or domain names. Consequently, novel character-based template generators are required that provide robust templates for any type of computer log data, which can be applied in security information and event management (SIEM) solutions, for continuous auditing, quality inspection and control. In this chapter, we propose a novel approach for computing character-based templates, which combines comparison-based methods and heuristics. To achieve this goal, we solve the problem of efficiently calculating a multi-line alignment for a group of log lines and compute an accurate approximation of the optimal character-based template.

Parts of this chapter have been published in [119].