Design of Power Operation and Maintenance Audit System Based on Hadoop

doi:10.3969/j.issn.1006-2475.2020.01.009

Abstract

Abstract: State Grid is becoming more and more informative. The amount of data generated by single-machine operation and maintenance audit system is increasing. The performance of high-efficiency storage analysis for massive data is seriously degraded, and system stability is reduced. In order to meet the requirement of State Grid for data storage analysis and system stability of the operation and maintenance audit system, based on the Hadoop open source architecture, the paper proposes a massive data distributed storage method based on Hadoop cluster and heartbeat detection technology based on Heartbeat, and designs a power operation and maintenance audit system based on Hadoop. The experimental results show that the availability of power operation and maintenance audit system is increased by 8.42% compared to stand-machine system. The performance of massive data for storage analysis is greatly improved. It has the advantages of stable system operation and uninterrupted service.

Key words: operation and maintenance audit, Hadoop, data storage, cluster, Heartbeat

CLC Number:

TP311.13

SU Lin-ping, AN Ran, LI Wei, CUI Wen-chao, ZHANG Xiao-liang. Design of Power Operation and Maintenance Audit System Based on Hadoop[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2020.01.009.

References

［1］ DAHDOUH K, DAKKAK A, OUGHDIR L, et al. Large-scale e-learning recommender system based on Spark and Hadoop［J/OL］. Journal of Big Data,2019(6).（2019-01-08）［2019-03-22］. https://doi.org/10.1186/s40537-019-0164-4.
［2］ BAKLI M, SAKR M, SOLIMAN T H A. Hadoop Trajectory: A Hadoop spatiotemporal data processing extension［J］. Journal of Geographical Systems, 2019,21(2):211-235.
［3］马刚. 朴素贝叶斯算法的改进与应用［D］. 合肥：安徽大学, 2018.
［4］滕志昊. 异构环境下Hadoop集群负载均衡策略的研究与实现［D］. 北京：北京邮电大学, 2018.
［5］樊程. 基于SOA的服务动态扩展机制研究与实现［D］. 青岛：青岛大学, 2013.
［6］王猛. 基于异构Hadoop集群的MapReduce任务调度算法研究［D］. 西安：西北大学, 2018.
［7］刘芳云,张志勇,李玉祥. 基于Hadoop的分布式并行增量爬虫技术研究［J］. 计算机测量与控制, 2018,26(10):269-275.
［8］肖佳,王古城,杨旭,等. Heartbeat心跳检测与负载均衡的设计与实现［J］.电子设计工程, 2015,23(18):176-179.
［9］李勇. 基于Linux的Heartbeat心跳检测系统研究与设计［J］. 计算机与数字工程, 2016,44(11):2293-2297.
［10］段海军,叶宏,张鹏. 面向IMA网络文件系统的双服务器结构设计［J］. 航空计算技术, 2012,42(1):92-95.
［11］张立航,陈大勇,刘海峰,等. 基于Linux的Heartbeat心跳检测系统设计［J］. 计算机安全, 2011(10):45-47.
［12］范绚春,游志胜,冯子亮,等. 一种FDP Heartbeat心跳检测方案［J］. 四川大学学报(自然科学版), 2004(6):1172-1174.
［13］李丽. 服务器集群技术探析［J］. 数字技术与应用, 2013(10):224.
［14］刘清华,杨桂芹. 基于Heartbeat+MySQL的Heartbeat心跳检测实现与改进［J］. 自动化与仪器仪表, 2015(1):159-160.
［15］徐超群. 高可用集群系统关键技术的研究与实现［D］. 南京：南京航空航天大学, 2014.
［16］梁烜彰. 基于大数据平台的用户搜索日志分析和研究［D］. 广州：华南理工大学, 2018.
［17］胡治宇,利莉. 基于Hadoop的高校网络舆情关键技术研究［J］. 计算机产品与流通, 2019(6):247-248.
［18］胡非,刘为,王丹妮,等. 运维安全审计在电网企业中的建设及应用［J］. 计算机产品与流通, 2018(12):267.
［19］姜婷. 基于应用虚拟化的运维审计系统的设计与实现［D］. 北京：华北电力大学(北京), 2018.
［20］杨健兵. 改进的K-means聚类算法在公交IC卡数据分析中的应用研究［J］. 软件工程, 2019,22(5):32-34.
［21］ANSARI Z, AFZAL A, SARDAR T H. Data categorization using Hadoop MapReduce-based parallel K-means clustering［J］. Journal of The Institution of Engineers (India): Series B, 2019,100(2):95-103.
［22］李士果,卢建云. Hadoop 3.0大数据平台性能［J］. 电子技术与软件工程, 2019(5):158-160.
［23］于波,李庆建,王卫. 实时通信集群负载均衡策略研究与应用［J］. 计算机系统应用, 2019,28(5):167-172.
［24］汪瑜斌. 基于Linux构建Hadoop高性能服务器集群［J］. 通信电源技术, 2018,35(9):151-152.
［25］曾德超,李晋,范兴丰. 基于Hadoop架构的企业知识服务平台设计与实现［J］. 自动化应用, 2019(1):151-153.

[1]	WANG Qiu-yi, ZHOU Hao, ZHENG Ting-ting. Improved RetinaNet Target Detection Method for Power Equipment [J]. Computer and Modernization, 2024, 0(01): 47-52.
[2]	YANG Bo, ZHUANG Yi. Fault Detection Method of Control Cluster Based on AOA-MSVM [J]. Computer and Modernization, 2023, 0(12): 112-116.
[3]	WANG Chong-yang, ZHUANG Yi. Load Balancing Algorithm of Multi-job Cluster Based on SDN and Improved CSA Algorithm [J]. Computer and Modernization, 2023, 0(11): 28-35.
[4]	WANG Hong-jie, XU Sheng-chao. Clustering Method of Cloud Platform Abnormal Transmission Data Based on Hilbert Similarity [J]. Computer and Modernization, 2023, 0(09): 27-31.
[5]	HAN Xue. Multi Path Planning Based on Constrained Clustering and Particle Swarm Optimization [J]. Computer and Modernization, 2023, 0(08): 7-11.
[6]	SUN Zi-yu, REN Ran, WEI Xi-zhe. Research on Stock Classification and Forecast Based on DTW-TCN [J]. Computer and Modernization, 2023, 0(08): 31-37.
[7]	ZHANG Jun, SU Wen-hao. Optimization Method of Hadoop File Archiving Based on LZO [J]. Computer and Modernization, 2023, 0(06): 1-6.
[8]	WANG Yi-cheng, ZHANG Guo-liang, ZHANG Zi-jie, . Small Object Detection Method Based on Improved YOLOv5 [J]. Computer and Modernization, 2023, 0(05): 100-105.
[9]	MA Yu-juan, HAN Jian-ning, SHI Shao-jie, CAO Shang-bin, YANG Zhi-xiu. Improved Kmeans Segmentation Algorithm for Brain Tumor Based on HMRF [J]. Computer and Modernization, 2023, 0(03): 1-5.
[10]	HONG Tao, ZHU Peng-yu, GUO Bo, WANG Jing-Yu. Communication Fault Diagnosis Knowledge Base Construction and Iteration Based on Semi-supervised Clustering [J]. Computer and Modernization, 2023, 0(02): 28-33.
[11]	LIU Xing-jian, YANG Xiao-fu, HU Lei. A Semi-supervised Model with Non-negative Matrix Factorization for Multiplex Network Clustering [J]. Computer and Modernization, 2023, 0(02): 83-88.
[12]	WEN Zi-xin, LI Shao-ying, WANG Bin-cheng, LIU Bo, . Face Clustering Method Based on Nearest Neighborhood Aggregation [J]. Computer and Modernization, 2022, 0(12): 81-87.
[13]	GUAN Yun-peng, LIU Yu-long. Categorical Data Clustering Based on Extraction of Associations from Co-association Matrix [J]. Computer and Modernization, 2022, 0(11): 1-8.
[14]	ZHANG Zi-xuan, SHA Xiu-yan, XIAO Fei, SU Bao-chan, SUI Yu-lu, MENG Zi-chen. Research and Application of Hesitant Fuzzy Canopy-K-means Clustering Algorithm [J]. Computer and Modernization, 2022, 0(11): 17-21.
[15]	DING Xu-dong, YANG Dong-run, LIU Hui, ZHAO Xing-kai, ZHANG Ying, SUN Mei, . A Data-driven Online Modeling Method for Evaporators [J]. Computer and Modernization, 2022, 0(11): 22-31.