Loglead-fast and integrated log loader, enhancer, and anomaly detector

MV Mäntylä, Y Wang, J Nyyssölä - 2024 IEEE International …, 2024 - ieeexplore.ieee.org
MV Mäntylä, Y Wang, J Nyyssölä
2024 IEEE International Conference on Software Analysis, Evolution …, 2024ieeexplore.ieee.org
This paper introduces LogLead, a tool designed for efficient log analysis benchmarking.
LogLead combines three essential steps in log processing: loading, enhancing, and
anomaly detection. The tool leverages Polars, a high-speed DataFrame library. We currently
have Loaders for eight systems that are publicly available (HDFS, Hadoop, BGL,
Thunderbird, Spirit, Liberty, TrainTicket, and GC Webshop). We have multiple enhancers
with three parsers (Drain, Spell, LenMa), Bert embedding creation and other log …
This paper introduces LogLead, a tool designed for efficient log analysis benchmarking. LogLead combines three essential steps in log processing: loading, enhancing, and anomaly detection. The tool leverages Polars, a high-speed DataFrame library. We currently have Loaders for eight systems that are publicly available (HDFS, Hadoop, BGL, Thunderbird, Spirit, Liberty, TrainTicket, and GC Webshop). We have multiple enhancers with three parsers (Drain, Spell, LenMa), Bert embedding creation and other log representation techniques like bag-of-words. LogLead integrates to five supervised and four unsupervised machine learning algorithms for anomaly detection from SKLearn. By integrating diverse datasets, log representation methods and anomaly detectors, LogLead facilitates comprehensive benchmarking in log analysis research. We show that log loading from raw file to dataframe is over 10x faster with LogLead compared to past solutions. We demonstrate roughly 2x improvement in Drain parsing speed by off-loading log message normalization to LogLead. Our brief benchmarking on HDFS indicates that log representations extending beyond the bag-of-words approach offer limited additional benefits. Tool URL: https://github.com/EvoTestOps/LogLead.
ieeexplore.ieee.org
以上显示的是最相近的搜索结果。 查看全部搜索结果