Ali Shan, Boufaied Chaima, Bianculli Domenico, Branco Paula, Briand Lionel
University of Ottawa, Ottawa, Canada.
Prince Sultan University, Riyadh, Saudi Arabia.
Empir Softw Eng. 2025;30(5):129. doi: 10.1007/s10664-025-10669-3. Epub 2025 Jun 23.
Growth in system complexity increases the need for automated techniques dedicated to different log analysis tasks such as Log-based Anomaly Detection (LAD). The latter has been widely addressed in the literature, mostly by means of a variety of deep learning techniques. However, despite their many advantages, that focus on deep learning techniques is somewhat arbitrary as traditional Machine Learning (ML) techniques may perform well in many cases, depending on the context and datasets. In the same vein, semi-supervised techniques deserve the same attention as supervised techniques since the former have clear practical advantages. Further, current evaluations mostly rely on the assessment of detection accuracy. However, this is not enough to decide whether or not a specific ML technique is suitable to address the LAD problem in a given context. Other aspects to consider include training and prediction times as well as the sensitivity to hyperparameter tuning, which in practice matters to engineers. In this paper, we present a comprehensive empirical study, in which we evaluate a wide array of supervised and semi-supervised, traditional and deep ML techniques w.r.t. four evaluation criteria: detection accuracy, time performance, sensitivity of detection accuracy and time performance to hyperparameter tuning. Our goal is to provide much stronger and comprehensive evidence regarding the relative advantages and drawbacks of alternative techniques for LAD. The experimental results show that supervised traditional and deep ML techniques fare similarly in terms of their detection accuracy and prediction time on most of the benchmark datasets considered in our study. Moreover, overall, sensitivity analysis to hyperparameter tuning with respect to detection accuracy shows that supervised traditional ML techniques are less sensitive than deep learning techniques. Further, semi-supervised techniques yield significantly worse detection accuracy than supervised techniques.
系统复杂性的增加使得对专门用于不同日志分析任务（如基于日志的异常检测（LAD））的自动化技术的需求也随之增加。后者在文献中已得到广泛探讨，主要是通过各种深度学习技术。然而，尽管深度学习技术有诸多优点，但仅关注深度学习技术在某种程度上是武断的，因为传统机器学习（ML）技术在许多情况下可能表现良好，这取决于具体的上下文和数据集。同样，半监督技术与监督技术一样值得关注，因为前者具有明显的实际优势。此外，当前的评估大多依赖于检测准确率的评估。然而，这不足以确定特定的ML技术在给定上下文中是否适合解决LAD问题。其他需要考虑的方面包括训练和预测时间以及对超参数调整的敏感性，而在实践中这些对工程师来说很重要。在本文中，我们进行了一项全面的实证研究，在研究中我们根据四个评估标准：检测准确率、时间性能、检测准确率和时间性能对超参数调整的敏感性，对一系列监督和半监督、传统和深度ML技术进行了评估。我们的目标是为LAD替代技术的相对优缺点提供更有力和全面的证据。实验结果表明，在我们研究中考虑的大多数基准数据集上，监督传统ML技术和深度ML技术在检测准确率和预测时间方面表现相似。此外，总体而言，关于检测准确率的超参数调整敏感性分析表明，监督传统ML技术比深度学习技术更不敏感。此外，半监督技术产生的检测准确率明显低于监督技术。
