LogOW: A semi-supervised log anomaly detection model in open-world setting

doi:10.1016/j.jss.2024.112305

	LogOW: A semi-supervised log anomaly detection model in open-world setting
	Ye, Jingwei 1; Liu, Chunbo 1; Gu, Zhaojun 1; Zhang, Zhikai 1; Meng, Xuying 2; Zhang, Weiyao 2; Zhang, Yujun 2
	2025-04-01
发表期刊	JOURNAL OF SYSTEMS AND SOFTWARE
ISSN	0164-1212
卷号	222 页码:13
摘要	Log anomaly detection is a method for finding abnormal behavior and faults in systems. However, existing methods face two main challenges: the open-world problem and the cold-start problem. The open-world problem means that the test set may contain new classes that are not in the training set, while the cold- start problem means that the initial training data are scarce, both for normal and abnormal log sequences. Most existing methods assume a closed-world setting and rely on sufficient normal data, which limits their adaptability to new log environments. We propose LogOW, a novel log anomaly detection model that can learn from a few normal log sequences. The model finds emerging normal log sequences in the open-world setting through the open-world sample retrieval module. Through the incremental pre-training module, these log sequences are fine-tuned in an online mode for model parameters. First, we train a basic model from normal log sequences using Masked-Language Modeling(MLM). During the testing phase, we then combine the anomaly score and the uncertainty score obtained through a novel dynamic multi-mask to distinguish closed-world normal log sequences from the test set. Next, we cluster the open-world log sequences based on fused sequence and count features, and identify the abnormal ones and the new normal ones. Finally, we update our model with the new normal sequences in the next time period. Experiments on three log datasets and real-world airport logs show that our model outperforms traditional models in the open-world and lack of training data setting.
关键词	Semi-supervised Log anomaly detection Open-world Uncertainty estimation Incremental pre-training Cold-start
DOI	10.1016/j.jss.2024.112305
收录类别	SCI
语种	英语
资助项目	National Science Foundation of China[U2333201] ; National Key R&D Program of China[2021YFF0603902] ; Civil Aviation Safety Capacity Building Foundation of China[PESA2022093] ; Civil Aviation Safety Capacity Building Foundation of China[PESA2023101] ; Pilot for Major Scientific Research Facility of Jiangsu Province of China[BM2021800] ; National Natural Science Foundation of China[62372429] ; Project on Cyber Security and Informatization of Chinese Academy of Sciences[CAS-WX2022SF-040]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering ; Computer Science, Theory & Methods
WOS记录号	WOS:001383422000001
出版者	ELSEVIER SCIENCE INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41075
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, Chunbo
作者单位	1.Civil Aviat Univ China, Tianjin 300300, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100089, Peoples R China
推荐引用方式 GB/T 7714	Ye, Jingwei,Liu, Chunbo,Gu, Zhaojun,et al. LogOW: A semi-supervised log anomaly detection model in open-world setting[J]. JOURNAL OF SYSTEMS AND SOFTWARE,2025,222:13.
APA	Ye, Jingwei.,Liu, Chunbo.,Gu, Zhaojun.,Zhang, Zhikai.,Meng, Xuying.,...&Zhang, Yujun.(2025).LogOW: A semi-supervised log anomaly detection model in open-world setting.JOURNAL OF SYSTEMS AND SOFTWARE,222,13.
MLA	Ye, Jingwei,et al."LogOW: A semi-supervised log anomaly detection model in open-world setting".JOURNAL OF SYSTEMS AND SOFTWARE 222(2025):13.