Load Balancing Technology Under Big Data Architecture Based on HBase

Abstract

Abstract: With the continuous growth of the scale and type of meteorological data, meteorological data has gradually entered the stage of massive service, and meanwhile providing more agile data services based on the background of big data has become an urgent demand for business development. In this paper, a load balancing algorithm and strategy based on HBase system is proposed for semi/unstructured meteorological data. In the actual test and comparison, it is found that the system can meet more than 2 million grid points and 100 concurrent scenarios, and the query speed is within 2 s. Compared with the load balancing algorithm which is not added load balance, the system data response speed is improved by 42.69 times, which can effectively meet the actual business needs.

Key words: load balance, HBase, distributed storage, big data, HDFS

LEI Ming, JIANG Han-sheng, WU Guo-liang, ZHAO Yu-juan, LIANG Jian. Load Balancing Technology Under Big Data Architecture Based on HBase[J]. Computer and Modernization, 2021, 0(06): 91-95.

References ［23］

［1］	邓维,刘方明,金海,等. 云计算数据中心的新能源应用：研究现状与趋势［J］. 计算机学报, 2013,36(3):582-598.
［2］	潘飞,蒋从锋,徐向华,等. 负载相关的虚拟机放置策略［J］. 小型微型计算机系统, 2013,34(3):520-524.
［3］	魏彦红. BI-PaaS平台中BI应用的调度方案的研宄与实现［D］. 北京:北京邮电大学, 2014.
［4］	BANAEE H, LOUTFI A. Data-driven rule mining and representation of temporal patterns in physiological sensor data［J］. IEEE Journal of Biomedical & Health Informatics, 2015,19(5):1557-1566.
［5］	陈作聪. 基于灰色模型的海洋大数据迁移算法设计［J］. 广东工业大学学报, 2018,35(3):95-99.
［6］	张栗粽,崔园,罗光春,等. 面向大数据分布式存储的动态负载均衡算法［J］. 计算机科学, 2017,44(5):178-183.
［7］	张丽梅. 基于负载均衡的云资源调度策略研究［D］. 银川:宁夏大学, 2014.
［8］	张令涛,赵林,张亮,等. 配用电大数据分布式计算集群负载均衡框架［J］. 电网技术, 2019,43(1):259-265.
［9］	章勇,张洁卉,柳斌. 全局负载均衡下云环境中的大数据动态迁移方法［J］. 计算机科学, 2018,45(1):196-199.
［10］	张晋芳,王清心,丁家满. 一种云计算环境下大数据动态迁移策略［J］. 计算机工程, 2016,42(5):13-17.
［11］	冯兴杰,贺阳. 改进的Hadoop作业调度算法［J］. 计算机工程与应用, 2017,53(12):85-91.
［12］	孙兰芳,张曦煌. 基于蜜蜂采蜜机理的云计算负载均衡策略［J］. 计算机应用研究, 2016,33(4):1179-1182.
［13］	GHADAH A, TAHIR B S, NADINE A, et al. Distributed dynamic load balancing in a heterogeneous network using LTE and TV white spaces［J］. Wireless Networks, 2015,21(7):2413-2424.
［14］	李梓杨,于炯,卞琛,等. 基于负载感知的数据流动态负载均衡策略［J］. 计算机应用, 2017,37(10):2760-2766.
［15］	鲁亮,于炯,卞琛,等. Storm环境下基于权重的任务调度算法［J］. 计算机应用, 2018,38(3):699-706.
［16］	孙大为,张广艳,郑纬,等. 大数据流式计算:关键技术及系统实例［J］. 软件学报, 2014,25(4):839-862.
［17］	张译天,于炯,鲁亮,等. 大数据流式计算框架Heron环境下的流分类任务调度策略［J］. 计算机应用, 2019,39(4):1106-1116.
［18］	张笑东,夏筱筠,吕海峰,等. 大数据网络并行计算环境中生理数据流动态负载均衡［J］. 吉林大学学报（工学版）, 2020,50(1):247-254.
［19］	蒋本立,张小平. 大数据网络的均衡调度平台设计与改进［J］. 现代电子技术, 2016,39(6):62-65.
［20］	冯兴杰,贺阳. 改进的Hadoop作业调度算法［J］. 计算机工程与应用, 2017,53(12):85-91.
［21］	伍湘君,陈德辉,宋君强,等. GRAPES全球格点模式的并行计算负载平衡策略［J］. 气象学报, 2010,68(5):591-597.
［22］	金之雁,王鼎兴. 一种在异构系统中实现负载平衡的方法［J］. 应用气象学报, 2003,14(4):410-418.
［23］	王亮,毛科峰,陈希. Gerris数值方案及其在海洋数值模拟中的应用［J］. 南京信息工程大学学报(自然科学版), 2015,7(1):92-96.

[1]	ZHANG Jun, SU Wen-hao. Optimization Method of Hadoop File Archiving Based on LZO [J]. Computer and Modernization, 2023, 0(06): 1-6.
[2]	ZHOU Ming-sheng, ZHANG Wen. A Smart Park Management Platform for Multi-source Data [J]. Computer and Modernization, 2023, 0(05): 68-74.
[3]	QIU Jin-shui, ZHUANG Hui-fu, JIN Tao. Design of Intelligent Retrieval System for Massive Plant Images [J]. Computer and Modernization, 2022, 0(10): 62-67.
[4]	SHAN Ke, ZHANG Yi-ming, LIU Rui-xia, . Research and Design of Science and Technology Service Resource Pool Oriented to Central Plains Urban Agglomeration [J]. Computer and Modernization, 2022, 0(07): 91-96.
[5]	HUANG AN-qi, MIAO Fang, YANG Wen-hui, NI Ya-ting, JIANG Yuan. Design of Structured Data Registration Engine Based on Data Architecture [J]. Computer and Modernization, 2022, 0(05): 82-89.
[6]	CAO Yu, LI Xiao-hui, LIU Zhong-lin, JIA He, FEI Zhi-wei. Review of Big Data Workflow Orchestration and Management System in Cloud Environment [J]. Computer and Modernization, 2022, 0(01): 41-53.
[7]	ZHANG Xiao-fang, FENG Hui-fang. Dynamic Optimal Path Planning Based on Trajectory Big Data [J]. Computer and Modernization, 2021, 0(11): 82-88.
[8]	LI Ming, CHEN Ji-fu, YI Xiao-rong, LIU Shu-ming. An Environment Monitoring System for Dongting Lake Based on JFinal Framework [J]. Computer and Modernization, 2021, 0(10): 41-48.
[9]	WEI Yun-dong. Intelligent Talent Recommendation Method Based on Big Data Technology [J]. Computer and Modernization, 2021, 0(07): 60-64.
[10]	LIU Feng, ZOU Chen-song, CUI Wei. Web Service Composition Based on K-medoids Point Optimization Algorithm in Big Data Environment [J]. Computer and Modernization, 2020, 0(12): 20-24.
[11]	PAN Wei-jun, LIU Hao-chen, WANG Run-dong, HU Bo-wen. Application of Improved Spark System Based on ANN in Big Data Processing of Air Traffic Management#br# [J]. Computer and Modernization, 2020, 0(12): 78-82.
[12]	ZHOU Xian-lai. Big Data Mining Algorithm of Heterogeneous Multi-core Platform Based on Semantic Segmentation [J]. Computer and Modernization, 2020, 0(10): 40-43.
[13]	WU Hai-wei, WANG Xiao-zhong, ZHU Fa-shun, . A Scheduling Method of Smart Grid Based on Genetic Algorithm [J]. Computer and Modernization, 2020, 0(09): 122-126.
[14]	XU Jian-peng, ZHANG Hui, WU Qiong, WANG Hui, WANG Bing. Design and Application of Big Data Platform of Anhui Meteorological Service for Agriculture [J]. Computer and Modernization, 2020, 0(08): 105-108.
[15]	LIU Zhang-rong. Dynamic Suppression Algorithm of Semi-distributed Zombie Network Based on Big Data [J]. Computer and Modernization, 2020, 0(08): 109-113.