Transaction Design and Implementation for HDFS Access Middleware

doi:10.3969/j.issn.1006-2475.2017.01.009

Abstract

Abstract: To settle the problem of Hadoop distributed file system (HDFS) not supporting transaction and difficult to recover from failure, a HDFS access middleware to support transaction for its application systems was proposed. Accessing to HDFS data, the solution can provide transaction supporting. Firstly, the transaction requirements for those applications systems accessing to HDFS were analyzed. Transaction process, transaction log and transaction recovery were described. Secondly, the transaction process and architecture of the middleware were designed. Thirdly, J2EE was used to implement the design of the proposed middleware. Finally, a test case was designed to test the proposed middleware design and implementation. It illustrates the proposed middleware can simplify both the process of accessing to HDFS and the process of recovering from failure.

Key words: HDFS, middleware, transaction, DFS, Hadoop

CLC Number:

TP311

LI Na, CHEN Zheng-ming, LYU Jia, LIU Chun-fang. Transaction Design and Implementation for HDFS Access Middleware[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2017.01.009.

References

[1] 张天宇,贺金鑫,王阳,等. 基于NoSQL数据库的地学大数据高效存储方法[J]. 吉林大学学报(信息科学版), 2013,31(6):604-608.

[2] 何颖鹏. 非结构化数据统一存储平台的设计与实现[D]. 杭州:浙江大学, 2013.

[3] Ghemawat S, Gobioff H, Leung S. File and storage systems: The Google file system[C]// Proceedings of the 9th ACM Symposium on Operating Systems Principles. 2003:29-43.

[4] Wu Sai, Chen Gang, Zhou Xianke, et al. PABIRS: A data access middleware for distributed file systems[C]// Proceedings of the 31st IEEE International Conference on Data Engineering. 2015:113-124.

[5] Gupta R, Gupta H, Gupta S, et al. A middleware for managing big-data flows[M]// Web Information Systems Engineering - WISE 2013. Springer Berlin Heidelberg, 2013:410-424.

[6] 廖振松,杨勤,曾志华. 基于移动信令数据分析的大数据中间件研究[J]. 信息通信, 2014(8):163-164.

[7] 宫海林,郭长国,苑洪亮,等. 具有事务能力的文件存储技术[J]. 计算机研究与发展, 2007,44(z1):64-68.

[8] MyCat开源项目组. MyCat权威指南[Z]. MyCat开源项目组, 2015.

[9] 卫洁. MapReduce框架下的贝叶斯文本分类学习研究[D]. 太原:山西财经大学, 2012.

[10] 彭辅权. Hadoop集群技术的优化与应用研究[D]. 杭州:浙江大学, 2012.

[11] 陆嘉恒. Hadoop实战[M]. 2版. 北京:机械工业出版社, 2011:158-178.

[12] 胡金金. 内存数据库若干关键技术的研究[D]. 镇江:江苏大学, 2012.

[13] 西尔伯沙茨. 数据库系统概念[M]. 5版. 北京:机械工业出版社, 2008:404-416.

[14] 李小龙. 分布式数据库事务支持[D]. 成都:电子科技大学, 2014.

[15] 肖欣欣. 基于J2EE的电力通信资源管理系统的设计与开发[D]. 苏州:苏州大学, 2008.

[16] 黄华. 基于J2EE平台的数据访问中间件的研究与实现[D]. 南京:河海大学, 2005.

[17] 朱跃龙. 基于反射的水利数据访问中间件技术研究[D]. 南京:河海大学, 2007.

[18] 曹鸣鹏,赵伟,许林英. J2EE技术及其实现[J]. 计算机应用, 2001,21(10):20-23.