计算机与现代化

• 软件工程 • 上一篇    下一篇

HDFS访问中间件的事务设计与实现

  

  1. (河海大学物联网工程学院,江苏 常州 213022)
  • 收稿日期:2016-06-16 出版日期:2017-01-12 发布日期:2017-01-11
  • 作者简介:李娜(1991-),女,湖南永州人,河海大学物联网工程学院硕士研究生,研究方向:分布式系统,软件工程; 陈正鸣(1965-),男,教授,博士,CFF高级会员,研究方向:大数据可视化与分析,数字化建模,虚拟现实与3D打印技术; 吕嘉(1977-),男,讲师,博士,研究方向:分布式系统,软件工程; 刘春芳(1990-),女,硕士研究生,研究方向:计算机图形学。
  • 基金资助:
    江苏省科技支撑计划项目(BE2014048)

Transaction Design and Implementation for HDFS Access Middleware

  1. (College of IOT Engineering, Hohai University, Changzhou 213022, China)
  • Received:2016-06-16 Online:2017-01-12 Published:2017-01-11

摘要: 针对Hadoop分布式文件系统(HDFS)不支持事务导致其故障恢复比较复杂的问题,提出一种支持事务的HDFS访问中间件的设计方案。该方案在对HDFS进行数据访问时还能提供事务支持。首先,分析访问HDFS的应用系统的事务需求,从HDFS事务流程、事务日志和事务恢复3个方面进行描述。其次,设计HDFS访问中间件的事务流程和体系架构。再次,根据设计方案,采用J2EE技术进行系统实现。最后,设计一个实例来测试中间件的设计和实现,通过这个实例说明,该中间件能有效地简化HDFS的数据访问和事务故障恢复的过程。

关键词: HDFS, 中间件, 事务, 分布式文件系统, Hadoop

Abstract: To settle the problem of Hadoop distributed file system (HDFS) not supporting transaction and difficult to recover from failure, a HDFS access middleware to support transaction for its application systems was proposed. Accessing to HDFS data, the solution can provide transaction supporting. Firstly, the transaction requirements for those applications systems accessing to HDFS were analyzed. Transaction process, transaction log and transaction recovery were described. Secondly, the transaction process and architecture of the middleware were designed. Thirdly, J2EE was used to implement the design of the proposed middleware. Finally, a test case was designed to test the proposed middleware design and implementation. It illustrates the proposed middleware can simplify both the process of accessing to HDFS and the process of recovering from failure.

Key words: HDFS, middleware, transaction, DFS, Hadoop

中图分类号: