Zhou Kun, Wang Wenyong, Hu Teng, Deng Kai
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
Institute for Computer Application, China Academy of Engineering Physics, Mianyang 621900, China.
Entropy (Basel). 2021 Feb 25;23(3):274. doi: 10.3390/e23030274.
Anomaly detection research was conducted traditionally using mathematical and statistical methods. This topic has been widely applied in many fields. Recently reinforcement learning has achieved exceptional successes in many areas such as the AlphaGo chess playing and video gaming etc. However, there were scarce researches applying reinforcement learning to the field of anomaly detection. This paper therefore aimed at proposing an adaptable asynchronous advantage actor-critic model of reinforcement learning to this field. The performances were evaluated and compared among classical machine learning and the generative adversarial model with variants. Basic principles of the related models were introduced firstly. Then problem definitions, modelling processes and testing were detailed. The proposed model differentiated the sequence and image from other anomalies by proposing appropriate neural networks of attention mechanism and convolutional network for the two kinds of anomalies, respectively. Finally, performances with classical models using public benchmark datasets (NSL-KDD, AWID and CICIDS-2017, DoHBrw-2020) were evaluated and compared. Experiments confirmed the effectiveness of the proposed model with the results indicating higher rewards and lower loss rates on the datasets during training and testing. The metrics of precision, recall rate and F1 score were higher than or at least comparable to the state-of-the-art models. We concluded the proposed model could outperform or at least achieve comparable results with the existing anomaly detection models.
异常检测研究传统上使用数学和统计方法进行。该主题已在许多领域广泛应用。最近，强化学习在诸如AlphaGo国际象棋比赛和视频游戏等许多领域取得了非凡的成功。然而，将强化学习应用于异常检测领域的研究却很少。因此，本文旨在提出一种适用于该领域的强化学习的自适应异步优势动作评判模型。对经典机器学习模型和具有变体的生成对抗模型的性能进行了评估和比较。首先介绍了相关模型的基本原理。然后详细介绍了问题定义、建模过程和测试。所提出的模型通过分别为两种异常情况提出适当的注意力机制神经网络和卷积网络，将序列和图像与其他异常情况区分开来。最后，使用公共基准数据集（NSL-KDD、AWID和CICIDS-2017、DoHBrw-2020）对经典模型的性能进行了评估和比较。实验证实了所提出模型的有效性，结果表明在训练和测试期间，该模型在数据集上具有更高的奖励和更低的损失率。精确率、召回率和F1分数等指标高于或至少与最先进的模型相当。我们得出结论，所提出的模型可以超越现有异常检测模型，或者至少取得与之相当的结果。
