Hadoop Speculation Execution Algorithm in Heterogeneous Environments

doi:10.3969/j.issn.1006-2475.2015.08.016

Abstract

Abstract: This article researches and analyzes the poor performance of the Hadoop speculation execution algorithm in heterogeneous environments, and puts forward a new improvement algorithm after researching source code deeply. The new algorithm can adjust the execution of backup task automatically to make it balanced according to system load condition, and get more precise stragglers queues using the way of putting the residual time value greater than 0.2 in task queue to judge the stragglers, based on the historical average completion time proposed by Zaharia. The new algorithm to a certain extent improves the performance of speculation execution in the heterogeneous environments.


Key words: Hadoop, performance optimization, speculation execution algorithm

CLC Number:

TP393.02

QI Peng-nian, ZHU Jin, HAO Jun-hui, XU Feng-ping. Hadoop Speculation Execution Algorithm in Heterogeneous Environments[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2015.08.016.

References

［1］ Apache Software Foundation. Hadoop 0.20.2 API［DB/OL］. http://hadoop.apache.org/common/docs, 2015-03-01.
［2］ Tom White. Hadoop权威指南［M］. 周敏奇,王晓玲,译. 2版. 北京:清华大学出版社， 2011.
［3］陈国良. 并行计算［M］. 北京:高等教育出版社， 2003.
［4］陆嘉恒. Hadoop实战Hadoop Action［M］. 北京:机械工业出版社， 2011.
［5］ Anderson T E, Culler D E, Patternson D A. A case for NOW［J］. IEEE Micro, 1995,15(1):54-64.
［6］李鑫. Hadoop框架的扩展和性能调优［D］. 西安:西安建筑科技大学， 2012.
［7］ Chuck Lam. Hadoop in Ation［M］. Manning Publications, 2010.
［8］曹英. 大数据环境下Hadoop性能优化的研究［D］. 大连:大连海事大学， 2013.
［9］赵书兰. 典型的Hadoop云计算［M］. 北京:电子工业出版社， 2013.
［10］韩晶. 大数据服务若干关键技术研究［D］. 北京:北京邮电大学, 2013.
［11］张密密. MapReduce模型在Hadoop实现中的性能分析及改进优化［D］. 成都:电子科技大学, 2010.
［12］沈案. 异构分布式系统中基于DVS的节能调度算法研究与实现［D］. 长沙:湖南大学, 2013.
［13］冯艳红. 实时多任务集成调度算法的研究［D］. 保定:华北电力大学, 2005.
［14］王峰. Hadoop集群作业的调度算法［J］. 程序员, 2009(12):119-121.
［15］周锋,李旭伟. 一种改进的MapReduce并行编程模型［J］. 科协论坛(下半月), 2009(2):65-66.
［16］孙广中,肖锋,熊曦. MapReduce模型的调度及容错机制研究［J］. 微电子学与计算机, 2007,24(9):178-180.
［17］曹廷. 一个异构多核调度算法及其实现［D］. 西安:西安电子科技大学, 2011.
［18］余利华. 分布式数据存储和处理的若干技术研究［D］. 杭州:浙江大学, 2008.
［19］王意洁,孙伟东,周松,等. 云计算环境下的分布存储关键技术［J］. 软件学报, 2012,23(4):962-986.
［20］李建江,崔健,王聃,等. MapReduce并行编程模型研究综述［J］. 电子学报, 2011,39(11):2635-2642.

[1]	ZHANG Jun, SU Wen-hao. Optimization Method of Hadoop File Archiving Based on LZO [J]. Computer and Modernization, 2023, 0(06): 1-6.
[2]	XU Xian-hui, WANG Shu-ying, ZENG Wen-qu. ElasticSearch Index Optimization Strategy for Engineering Data Retrieval [J]. Computer and Modernization, 2022, 0(02): 79-84.
[3]	LIU Lu, SHEN Guo-wei, GUO Chun, CUI Yun-he, JIANG Chao-hui, WU Da-yong. A Spark Streaming Parameter Optimization Method Based on Deep Reinforcement Learning [J]. Computer and Modernization, 2021, 0(10): 49-56.
[4]	SU Lin-ping, AN Ran, LI Wei, CUI Wen-chao, ZHANG Xiao-liang. Design of Power Operation and Maintenance Audit System Based on Hadoop [J]. Computer and Modernization, 2020, 0(01): 49-.
[5]	ZHOU Tian-qi. Spatial Distribution Statistics of Urban Population Based on Mobile Communication Big Data [J]. Computer and Modernization, 2018, 0(05): 45-.
[6]	DONG Xue， ZHANG De-ping. Global Sensitivity Analysis and Optimization of Submarine Combat Effectiveness #br# Based on Extreme Learning Machine [J]. Computer and Modernization, 2018, 0(05): 86-.
[7]	WEN He-ping1, BAO Jing-jing2, KE Ju-xin1, LIU Shu-wei1. Design of Hyperchaotic Lorenz Encryption Algorithm Based on Hadoop [J]. Computer and Modernization, 2018, 0(03): 108-.
[8]	CHU Jia-qi1,2, LIU Cong-jun1,2. Application of WeChat Public Service Platform in E-government [J]. Computer and Modernization, 2018, 0(02): 33-.
[9]	ZHU Shan, AI Li-hua. Large-scale Image Storage and Retrieval Based on Hadoop [J]. Computer and Modernization, 2017, 0(6): 61-66+83.
[10]	LI Cheng, CHAI Xiao-li, XIE Bin, TANG Peng. A Resource Scheduling Mechanism of Hadoop YARN [J]. Computer and Modernization, 2017, 0(11): 29-34.
[11]	ZHANG Jin, FENG Jun, LU Jia-min. Spatial Keyword Index Method Based on Hadoop [J]. Computer and Modernization, 2017, 0(11): 76-83.
[12]	FENG Jun, XU Wei-gang, FENG Du-qing, LU Jia-min, XU Huan. Research on Index Method of Massive Hydrology Data [J]. Computer and Modernization, 2017, 0(10): 29-35,41.
[13]	LI Na, CHEN Zheng-ming, LYU Jia, LIU Chun-fang. Transaction Design and Implementation for HDFS Access Middleware [J]. Computer and Modernization, 2017, 0(1): 46-50.
[14]	ZHANG Yan-nian . Wireless Channel Back-off Algorithm Optimization Design #br# Based on RBF Neural Network [J]. Computer and Modernization, 2016, 0(12): 16-21.
[15]	SUN Li-hua1, HU Mu1, MENG Qing-qiang1, QIAN Ya-kang1, WONG Song2. Solution for High Performance of Distribution Network Line Loss Big Data [J]. Computer and Modernization, 2016, 0(12): 42-46,50.