LDA-IDF Process Log Anomaly Detection Method for Smart Grids

doi:10.3969/j.issn.1006-2475.2025.06.018

Abstract

Abstract: Abstract: With the expansion of smart grid systems， the process log data has grown exponentially， making traditional log anomaly detection methods inadequate for handling such massive and complex data volumes. To promptly detect abnormal processes in smart grid systems and ensure their safe and stable operation， this paper proposes LogLDAIDF， a process log anomaly detection method based on LDA-IDF. The method first extracts LDA topic features from process log sequences， then introduces the IDF （Inverse Document Frequency） method to weight the topic features， and selects the top-k weighted topics as the final process log topic features， finally combining with a Bi-LSTM deep learning model for process log anomaly detection. Experimental results on two real-world datasets， HDFS and OpenStack， demonstrate that our proposed method significantly outperforms existing methods， achieving F1 scores of 0.987 and 0.969， respectively， results show its effectiveness and practicality in process log anomaly detection.

Key words: Key words: smart grid, anomaly detection, process log, deep learning, semantic feature

CLC Number:

TP309

LIU Shaojun, SHA Yitian, JIN Qianqian, CHEN Peng, WU Yue. LDA-IDF Process Log Anomaly Detection Method for Smart Grids[J]. Computer and Modernization, 2025, 0(06): 114-119.

References

［1］ YADAV R B， KUMAR P S， DHAVALE S V. A survey on log anomaly detection using deep learning［C］// 2020 8th International Conference on Reliability， Infocom Technologies and Optimization （Trends and Future Directions）. IEEE， 2020:1215-1220.
［2］ BLEI D M， NG A Y， JORDAN M I. Latent dirichlet allocation［J］. The Journal of Machine Learning Research， 2003，3:993-1022.
［3］ JIANG Z M， HASSAN A E， FLORA P， et al. Abstracting execution logs to execution events for enterprise applications （short paper）［C］// 2008 The Eighth International Conference on Quality Software. IEEE， 2008:181-186.
［4］ TANG L， LI T， PERNG C S. LogSig: Generating system events from raw textual logs［C］// Proceedings of the 20th ACM International Conference on Information and Knowledge Management. ACM， 2011:785-794.
［5］ HAMOONI H， DEBNATH B， Xu J， et al. Logmine: Fast pattern recognition for log analytics［C］// Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM， 2016:1573-1582.
［6］ HE P， ZHU J， ZHENG Z， et al. Drain: An online log parsing approach with fixed depth tree［C］// 2017 IEEE International Conference on Web Services. IEEE， 2017:33-40.
［7］ MESSAOUDI S， PANICHELLA A， BIANCULLI D， et al. A search-based approach for accurate identification of log message formats［C］// Proceedings of the 26th Conference on Program Comprehension. ACM， 2018:167-177.
［8］ DU M， LI F， ZHENG G， et al. Deeplog: Anomaly detection and diagnosis from system logs through deep learning［C］// Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM， 2017:1285-1298.
［9］ MIKOLOV T， SUTSKEVER I， CHEN K， et al. Distributed representations of words and phrases and their compositionality［J］. Advances in Neural Information Processing Systems， 2013，23：3111-3119.
［10］ SALTON G， BUCKLEY C. Term-weighting approaches in automatic text retrieval［J］. Information Processing & Management， 1988，24（5）:513-523.
［11］ DEVLIN J， CHANG M W， LEE K， et al. Bert: Pre-training of deep bidirectional transformers for language understanding［J］. arXiv preprint arXiv:1810.04805， 2018.
［12］ WANG M Y， XU L L， GUO L L. Anomaly detection of system logs based on natural language processing and deep learning［C］// 2018 4th International Conference on Frontiers of Signal Processing. IEEE， 2018:140-144.
［13］ AKANLE M， ADETIBA E， AKANDE V， et al. Experimentations with openstack system logs and support vector machine for an anomaly detection model in a private cloud infrastructure［C］// 2020 International Conference on Artificial Intelligence， Big Data， Computing and Data Communication Systems. IEEE， 2020:1-7.
［14］ LIU X， LIU W Y， DI X Q， et al. LogNADS: Network anomaly detection scheme based on log semantics representation［J］. Future Generation Computer Systems， 2021，124:390-405.
［15］ MENG W B， LIU Y， ZHU Y C， et al. Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs［C］// Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. IJCAI. 2019，19（7）:4739-4745.
［16］ MENG W B， LIU Y， ZHANG S L， et al. Logclass: Anomalous log identification and classification with partial labels［J］. IEEE Transactions on Network and Service Management， 2021，18（2）:1870-1884.
［17］ ZHANG X， XU Y， LIN Q W， et al. Robust log-based anomaly detection on unstable log data［C］// Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM， 2019:807-817.
［18］ GUO H X， YUAN S H， WU X T. Logbert: Log anomaly detection via bert［C］// 2021 International Joint Conference on Neural Networks. IEEE， 2021:1-8.
［19］ CHEN S， LIAO H. Bert-log: Anomaly detection for system logs based on pre-trained language model［J］. Applied Artificial Intelligence， 2022，36（1）. DOI: 10.1080/08839514.
2022.2145642.
［20］ AUSSEL N， PETETIN Y， CHABRIDON S. Improving performances of log mining for anomaly prediction through nlp-based log parsing［C］// 2018 IEEE 26th International Symposium on Modeling， Analysis， and Simulation of Computer and Telecommunication Systems. IEEE， 2018:237-243.
［21］ SATPATHI S， DEB S， SRIKANT R， et al. Learning latent events from network message logs［J］. IEEE/ACM Transactions on Networking， 2019，27（4）:1728-1741.
［22］ PETTINATO M， GIL J P， GALEAS P， et al. Log mining to re-construct system behavior: An exploratory study on a large telescope system［J］. Information and Software Technology， 2019，114:121-136.
［23］ OTOMO K， KOBAYASHI S， FUKUDA K， et al. Latent semantics approach for network log analysis: Modeling and its application［C］// 2021 IFIP/IEEE International Symposium on Integrated Network Management. IEEE， 2021:215-223.
［24］ NAPOLITANO D. Log analysis: Topic modeling applications on fine-features data processing system［D］. Turin：Politecnico di Torino， 2022.
［25］周建国，戴华，杨庚，等. 基于并列GRU分类模型的日志异常检测方法［J］. 南京理工大学学报， 2022，46（2）:198-204.
［26］孙雪奎，戴华，周建国，等. 基于日志模板主题特征的日志异常检测［J］. 计算机科学， 2023，50（6）:313-321.
［27］ HUANG Z H， XU W， YU K. Bidirectional LSTM-CRF models for sequence tagging［J］. arXiv preprint arXiv:1508.01991， 2015.
［28］ HE S， ZHU J， HE P， et al. Loghub: A large collection of system log datasets towards automated log analytics［J］. arXiv preprint arXiv:2008.06448， 2020.
［29］ ZHU J M， HE S L， LIU J Y， et al. Tools and benchmarks for automated log parsing［C］// 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice. IEEE，2019:121-130.
［30］周志华. 机器学习［M］. 北京:清华大学出版社， 2018.

[1]	LI Zhuoqi, ZHAO Lihui. Image Encryption Method Based on Poisoning Attack Strategy [J]. Computer and Modernization, 2025, 0(05): 41-47.
[2]	WANG Dongfang1, YANG Yan1, ZHANG Dong1, HAN Wenrui2, LI Mingchang2. DSA De-artifacting Algorithm Based on Deformation Field Registration [J]. Computer and Modernization, 2025, 0(05): 86-90.
[3]	WEI Yunsong1, 2, LI Jiaqiang1, 2, HE Chao1, 2, 3, YU Haisheng1, 2, CHEN Yanlin1, 2, ZHAO Longqing1, 2, WEI Rongkun1, 2. Research Advances on 3D Object Detection Method Based on Visual Information and LiDAR for Intelligent Driving [J]. Computer and Modernization, 2025, 0(05): 91-102.
[4]	XU Ling1, ZHANG Dong1, WEN Shen1, HU Ping2. Glioma Segmentation and Classification Network Assisted by Object Detection [J]. Computer and Modernization, 2025, 0(05): 111-116.
[5]	ZHUANG Yu, FU Xiaojin, LI Sha, WU Zheng. UAV Small Target Detection Based on XMB-YOLOv5s [J]. Computer and Modernization, 2025, 0(04): 29-35.
[6]	LI Kai, JIN Yunpeng, LI Haiyang, KONG Shasha, YANG Peng, FANG Chengwu, HUANG Xiangjie, HAN Yaosheng, LI Chunmei. AGP Calculation Methods in UAV Imagery Based on Image Segmentation [J]. Computer and Modernization, 2025, 0(04): 83-88.
[7]	WANG Jiale, SONG Wenai, FU Lizhen. Occupational Pneumoconiosis Screening Based on HA-Net Model [J]. Computer and Modernization, 2025, 0(04): 103-110.
[8]	TANG Rui1, WU Jianchao1, CHEN Jianbo1, CHAI Jiang1, WANG Qian1, HE Yuchen2. Improved YOLOv8s Algorithm Based on GiraffeDet for Transmission Line Icing Detection [J]. Computer and Modernization, 2025, 0(03): 6-11.
[9]	LI Haoran1, HE Wenxue1, XU Jiazhen1, YANG Banghua2. Classification Method of EEG Signals for Depression Based on Multi-Scale Dynamic Convolution and Attention Mechanism [J]. Computer and Modernization, 2025, 0(03): 60-65.
[10]	PU Yaya, WANG Yanbo, SU Yongdong, XU Zhongcheng. Multi-scale Feature Image Defogging Algorithm Based on Content-guided Attention Fusion [J]. Computer and Modernization, 2025, 0(03): 78-85.
[11]	LUO Hao, LI Xianfeng. Remote Sensing Image Classification Based on Multi-scale Feature Extraction [J]. Computer and Modernization, 2025, 0(03): 86-92.
[12]	ZHANG Yue, GUO Zixin, HUANG Yibin, YAN Tao. Network Intrusion Detection Method Based on Convolutional Neural Networks with convLSTM [J]. Computer and Modernization, 2025, 0(03): 119-126.
[13]	LIU Chongyi, LI Hua, REN Dejun, LIU Yaokai, WANG Yulong. Anomaly Detection Algorithm Based on Bidirectional Multi-scale Knowledge Distillation [J]. Computer and Modernization, 2025, 0(02): 58-63.
[14]	HE Guotao1, ZHAO Chunhui2, LIU Zhenyu1, WANG Long1. Optimization for Camera Self-calibration Based on Horizon Detection in Road Scenes [J]. Computer and Modernization, 2025, 0(02): 77-85.
[15]	ZHAO Yin, YIN Siqing, ZHANG Yonglai. Improved Traffic Sign Detection Algorithm of YOLOv7 [J]. Computer and Modernization, 2025, 0(02): 94-99.