Dynamic Replicas Strategy Based on Predicted Popularity

Abstract

Abstract:

To improve data availability and performance of cluster, current HDFS adapt uniform data replication. However, different files have different popularity and sometimes

the disparity is enormous, access to high popular data may hurt job performance. To address this problem, a dynamic replicas strategy based on predicted popularity is put

forward. By making full use of the recent data popularity, based on grey prediction model, we use Markov prediction model to correct the predicted deviation because of the burst

access and shifting access, and get the accurate predicted popularity of file. After then, finite channel service model based on the predicted popularity is established to

calculate the minimum replicas meeting user demand. Experimental result shows that compared with default data replication, our strategy can more effectively avoid contentions,

reduce the time consuming of job, and alleviated the network traffic.

Key words: high popular data, replica management, cloud computing, Hadoop, grey prediction, birth and death process

CLC Number:

TP274

ZHANG Song1， DU Qingwei1， SUN Jing2， SUN Zhen2. Dynamic Replicas Strategy Based on Predicted Popularity[J]. Computer and Modernization.

References

［1］
Wei Qingsong, Veeravalli B, Gong Bozhao, et al. CDRM: A costeffective dynamic replication management scheme for cloud storage cluster［C］// IEEE International Conference on

Cluster Computing(CLUSTER). 2010:188196.
［2］王意洁,孙伟东,周松,等. 云计算环境下的分布存储关键技术［J］. 软件学报, 2012，23(4):962986.
［3］White Tom. Hadoop: The Definitive Guide［M］. O’Reilly Media, Inc., 2009.

［4］Kala Karun A, Chitharanjan K. A review on Hadoop—HDFS infrastructure extensions［C］// IEEE Conference on Information & Communication Technologies(ICT). 2013:132

137.
［5］Ananthanarayanan G, Agarwal S, Kandula S, et al. Scarlett: Coping with skewed content popularity in mapreduce clusters［C］// Proceedings of the 6th ACM Conference on

Computer Systems. 2011:287300.
［6］Kousiouris G, Vafiadis G, Varvarigou T. Enabling proactive data management in virtualized Hadoop clusters based on predicted data activity patterns［C］// IEEE 8th

International Conference on P2P, Parallel, Grid, Cloud and Internet Computing(3PGCIC). 2013:18.
［7］孙大为,常桂然,高尚,等. Modeling a dynamic data replication strategy to increase system availability in cloud computing environments［J］. Journal of Computer Science and

Technology, 2012，27(2):256272.
［8］Abad C L, Lu Y, Campbell R H. DARE: Adaptive data replication for efficient cluster scheduling［C］// IEEE International Conference on Cluster Computing (CLUSTER). 2011:159

168.
［9］Wang Zhe, Li Tao, Xiong Naiyue, et al. A novel dynamic network data replication scheme based on historical access record and proactive deletion［J］. The Journal of

Supercomputing, 2012,62(1):227250.
［10］Shen Chunhui, Lu Weiming, Wu Jiangqin, et al. A digital library architecture supporting massive small files and efficient replica maintenance［C］// Proceedings of the

10th Annual Joint Conference on Digital Libraries. 2010:391392.
［11］Li Wendong, Yang Yun, Yuan Dong. A novel costeffective dynamic data replication strategy for reliability in cloud data centres［C］// IEEE 9th International Conference

on Dependable, Autonomic and Secure Computing(DASC). 2011:496502.
［12］Khaneghah E M, Mirtaheri S L, Grandinetti L, et al. A Dynamic Replication Mechanism to Reduce ResponseTime of I/O Operations in High Performance Computing Clusters［C

］// IEEE International Conference on Social Computing(SocialCom). 2013:738743.
［13］陶永才,张宁宁,石磊,等. 异构环境下云计算数据副本动态管理研究［J］. 小型微型计算机系统, 2013,34(7):14871492.
［14］夏怒,李伟,罗军舟,等. 一种基于波动类型识别的路由节点行为预测算法［J］. 计算机学报, 2014,37(2):326334.
［15］温祥西,孟相如,马志强,等. 小时间尺度网络流量混沌性分析及趋势预测［J］. 电子学报, 2012,40(8):16091616.
［16］周开利,康耀红. 神经网络模型及其MATLAB仿真程序设计［M］. 北京:清华大学出版社, 2005.
［17］Box G E, Jenkins G M, Reinsel G C. Time Series Analysis: Forecasting and Control［M］. John Wiley & Sons, 2013.

[1]	HE Yu-peng, TAO Yong, WANG Bing-heng, ZHAO Ying-nan. Research Status and Prospect of Edge Computing in Smart Distribution Network [J]. Computer and Modernization, 2023, 0(08): 87-92.
[2]	YANG Bo, XU Sheng-chao. Security Protection Method of Cloud Network Special Line Scenarios Based on SRv6#br# Service Chain [J]. Computer and Modernization, 2023, 0(08): 107-111.
[3]	ZHANG Jun, SU Wen-hao. Optimization Method of Hadoop File Archiving Based on LZO [J]. Computer and Modernization, 2023, 0(06): 1-6.
[4]	MAO Ming-yang, XU Sheng-chao. Communication Terminal Attack Behavior Identification Algorithm Based on Trusted Cloud Computing#br# [J]. Computer and Modernization, 2022, 0(11): 37-42.
[5]	CAO Yu, LI Xiao-hui, LIU Zhong-lin, JIA He, FEI Zhi-wei. Review of Big Data Workflow Orchestration and Management System in Cloud Environment [J]. Computer and Modernization, 2022, 0(01): 41-53.
[6]	ZHANG Xiao-min. An Attribute-based Multi-keyword Searchable Scheme Based on Bloom Filters [J]. Computer and Modernization, 2021, 0(08): 104-111.
[7]	DENG Bin-tao, XU Sheng-chao. A Differential Evolution K-mediods Clustering Algorithm Based on Dynamic Gemini Population [J]. Computer and Modernization, 2021, 0(07): 54-59.
[8]	WU Jun-cai, LIU Xue-fang . Digital Resource Allocation Based on Cloud Computing and Deep Belief Network [J]. Computer and Modernization, 2021, 0(05): 78-82.
[9]	ZHANG Li-jie, SHA Xiu-yan, YIN Chuan-cun, DUAN Jun-tao, ZHANG Xin-yi, LI Zi-tong, JIANG Fu-lei. Improved GM (1,1) Grey Prediction Model Based on Background Value of Variable Weight Optimization and Its Application [J]. Computer and Modernization, 2021, 0(01): 1-6.
[10]	LU Ge-hao, YANG Dan-ni. Development Status and Prospect of Blockchain as a Service [J]. Computer and Modernization, 2020, 0(10): 31-35.
[11]	SU Lin-ping, AN Ran, LI Wei, CUI Wen-chao, ZHANG Xiao-liang. Design of Power Operation and Maintenance Audit System Based on Hadoop [J]. Computer and Modernization, 2020, 0(01): 49-.
[12]	LIANG Ci1, CHEN Shi-ping2. Packet Drift Strategy Based on Packet Cluster Mapping Architecture [J]. Computer and Modernization, 2019, 0(11): 112-.
[13]	DING Shun1, CHEN Shi-ping2. Construction and Application of Experimental Platform #br# Based on Packet Cluster Mapping Mechanism [J]. Computer and Modernization, 2019, 0(11): 120-.
[14]	WEN Ting-ting, LI Hong-zhe. Elastic Load Balancing Algorithm for Cloud Computing Service [J]. Computer and Modernization, 2019, 0(10): 28-.
[15]	YUAN Min-fu, LI Yin， CHEN Sheng-jian， ZHENG Xiang-wei. Cloud platform Based Blockchain Networking and Data Storage Mechanism [J]. Computer and Modernization, 2019, 0(09): 46-.