大数据查询综述

doi:10.3969/j.issn.1006-2475.2017.04.017

摘要/Abstract

摘要： 大数据具有传统数据所不具有的数量大、种类多、速度快、真实性等特点，传统的数据查询技术满足不了日益增长的大数据查询需求，大数据查询技术应运而生并迅速发展。从大数据查询的角度出发，着重分析大数据存储技术、大数据处理平台以及大数据查询引擎等。分别对比介绍传统关系型数据库、NoSQL、NewSQL和它们在大数据查询处理上的应用，介绍当前流行的大数据处理平台以及在这些平台上运行的大数据查询引擎，对其优缺点进行了综合阐述。


关键词: 大数据, 查询, 存储, 处理平台, 查询引擎

Abstract: Big data has the characteristics such as summed up to volume, variety, velocity and veracity while traditional data doesnt have. As traditional technology doesnt meet the big data query, big data query technology has arose at the historic moment and was maturing rapidly. From the perspective of big data query, this paper analyzed the big data storage technology, big data processing platform and big data query engine, etc. This paper introduced traditional relational database, NoSQL, NewSQL and their application on big data query processing. Also, the current popular big data processing platforms and the big data query engines running on them were introduced, and an overview about their advantages and disadvantages was made.


Key words: big data, query, storage, processing platform, query engine

中图分类号:

TP301

张玉杰,于双元. 大数据查询综述[J]. 计算机与现代化, doi: 10.3969/j.issn.1006-2475.2017.04.017.

ZHANG Yu-jie, YU Shuang-yuan. Overview on Big Data Query[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2017.04.017.

参考文献

［1］徐保民. 云计算解密:技术原理及应用实践［M］. 北京:电子工业出版社, 2014.
［2］李雷. 大数据环境下数据存储与查询的研究［D］. 哈尔滨:哈尔滨工业大学, 2014.
［3］杜晓东. 大数据环境下基于Hbase的分布式查询优化研究［J］. 计算机光盘软件与应用, 2014(8):22-24.
［4］杨晶,刘天时,马刚. 分布式数据库数据分片与分配［J］. 现代电子技术, 2006,29(18):119-121.
［5］丁原,刘玉树,朱天焕. 多节点集群服务器系统共享磁盘私有网的研究［J］. 北京理工大学学报, 2001,21(1):62-64.
［6］张旭中. 分布式数据库查询优化技术［D］. 成都:电子科技大学, 2003.
［7］阮梦黎. 大数据挑战下的NoSQL系统研究［J］. 聊城大学学报(自然科学版), 2015,28(1):88-93.
［8］张滨,陈吉荣,乐嘉锦. 大数据管理技术研究综述［J］. 计算机应用与软件, 2014,31(11):1-5.
［9］ Pavlo A, Paulson E, Rasin A, et al. A comparison of approaches to large-scale data analysis［C］// ACM SIGMOD International Conference on Management of Data. 2009:165-178.
［10］Dean J, Ghemawat S. MapReduce: A flexible data processing tool［J］. Communications of the ACM, 2010,53(1):72-77.
［11］Stonebraker M, Abadi D, DeWitt D J, et al. MapReduce and parallel DBMSs: Friends or foes?［J］. Communications of the ACM, 2010,53(1):64-71.
［12］Olson M A, Bostic K, Seltzer M. Berkeley DB［C］// Freenix Track: 1999 Usenix Technical Conference. 1999:183-191.
［13］百度百科. Cassandra（开源分布式NoSQL数据库系统）［EB/OL］. http://baike.baidu.com/link?url=rgH8lTQ H4KpAWbVvCkh6oxtoUA6UoGlQDdVRhOuW3NzblbdZZW Yb4KI2bz8aaorZ63MD-V9J6Yl4_b7jzHrojEuSkGq7DTY84 qDAC2e5z-K, 2016-06-14.
［14］Neo Technology Inc. What Is Neo4j?［EB/OL］. https://neo4j.com/, 2016-06-14.
［15］百度百科. HyperGraphDB ［EB/OL］. http://baike.baidu.com/view/9329210.htm, 2016-08-14.
［16］Gilbert S, Lynch N. Brewers conjecture and the feasibility of consistent, available, partition-tolerant Web services［J］. ACM SIGACT News, 2002,33(2):51-59.
［17］Brewer E. CAP twelve years later: How the “Rules” have changed［J］. Computer, 2012,45(2):23-29.
［18］〖KG-*6〗Pavlo A, Aslett M. Whats really new with NewSQL?［J］. ACM SIGMOD Record, 2016,45(2):45-55.
［19］Corbett J C, Dean J, Epstein M, et al. Spanner: Googles globally-distributed database［C］// 10th Usenix Conference on Operating Systems Design and Implementation. 2012:251-264.
［20］VoltDB Inc.. What Is VoltDB? ［EB/OL］. https://www.voltdb.com/overview, 2013-09-29.
［21］Clustrix Inc.. A New Approach: Clustrix Sierra Database Engine［EB/OL］. http://www.clustrix.com/wp-content/uploads/2013/10/Clustrix_A-New-Approach_WhitePaper.pdf, 2013-09-29.
［22］NuoDB Greenbook Publication. NuoDB Emergent Architecture［EB/OL］. http://go.nuodb.com/rs/nuodb/images/Greenbook_Final.pdf, 2013-09-29.
［23］Grolinger K, Higashino W A, Tiwari A, et al. Data management in cloud environments: NoSQL and NewSQL data stores［J］. Journal of Cloud Computing: Advances Systems & Applications, 2013,2(1):Article No. 49.
［24］Ong K W, Papakonstantinou Y, Vernoux R. The SQL+〖KG-*3〗+ Semi-structured Data Model and Query Language: A Capabilities Survey of SQL-on-Hadoop, NoSQL and NewSQL 〖JP2〗Databases［EB/OL］. https://arxiv.org/abs/1405.3631v1, 2014-03-14.
［25］Niazi S, Ismail M, Grohsschmiedt S, et al. HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases［EB/OL］. https://arxiv.org/abs/1606.01588, 2016-06-06.
［26］Yuan Liyan, Wu Lengdong, You Jiahuai, et al. A demonstration of Rubato DB: A highly scalable newSQL database system for OLTP and big data applications［C］// Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. 2015:907-912.
［27］Doshi K A, Zhong Tao, Lu Zhongyan, et al. Blending SQL and newSQL approaches: Reference architectures for enterprise big data challenges［C］// Proceedings of the 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. 2013:163-170.
［28］Kepner J, Gadepally V, Hutchison D, et al. Associative Array Model of SQL, NoSQL, and NewSQL Databases［EB/OL］. https://arxiv.org/abs/1606.05797, 2016-06-18.
［29］Moniruzzaman A B M. NewSQL: Towards next-generation scalable RDBMS for Online Transaction Processing(OLTP) for big data management［J］. Inernational Journal of Database Theory and Application, 2014,7(6):121-130.
［30］黄宜华,苗凯翔. 深入理解大数据［M］. 北京:机械工业出版社, 2014.
［31］Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters［C］// Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation. 2004:107-113.
［32］Ghemawat S, Gobioff H, Leung S T. The Google file system［C］// Proceedings of the 19th ACM Symposium on Operating Systems Principles. 2003:29-43
［33］Chang F, Dean J, Ghemawat S, er al. Bigtable: A distributed storage system for structured data［C］// Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation. 2006:205-218.
［34］White T. Hadoop权威指南［M］. 华东师范大学数据科学与工程学院译. 北京:清华大学出版社, 2015.
［35］DeWitt D J, Stonebraker M. MapReduce: A Major Step Backwards［EB/OL］. http://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html, 2008-01-17.
［36］谌超,强保华,石龙. 基于Hadoop MapReduce的大规模数据索引构建与集群性能分析［J］. 桂林电子科技大学学报, 2012,32(4):307-312.
［37］Murthy A C. The Next Generation of Apache Hadoop MapReduce［EB/OL］. http://www.360doc.com/content/11/0321/22/2459_103337406.shtml, 2011-03-16.
［38］Vavilapalli V K, Murthy A C, Douglas C, et al. Apache Hadoop YARN: Yet another resource negotiator［C］// Proceedings of the 4th Annual Symposium on Cloud Computing. 2013:Article No. 5.
［39］JIRA. Browse Projects［EB/OL］. https://issues.apache.org/jira/secure/BrowseProjects.jspa#10292, 2016-06-01.
［40］数盟社区. 呼之欲出！比Spark快10倍的Hadoop3.0有哪些实用新特性？［EB/OL］. http://toutiao.com/i6290843711718818306/, 2016-05-31
［41］夏俊鸾,刘旭辉,邵赛赛,等. Spark大数据处理技术［M］. 北京:电子工业出版社, 2015.
［42］Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing［C］// Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. 2012:2.
［43］Zaharia M, Chowdhury M, Franklin M J, et al. Spark: Cluster computing with working sets［C］// Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. 2010:10.
［44］Zaharia M. An Architecture for Fast and General Data Processing on Large Clusters［M］. Association for Computing Machinery and Morgan & Claypool, 2016.
［45］于俊,向海,代其锋,等. Spark核心技术与高级应用［M］. 北京:机械工业出版社, 2016.
［46］Thusoo A, Sarma J S, Jain N, et al. Hive: A warehousing solution over a Map-Reduce framework［J］. Proceedings of the VLDB Endowment, 2011,2(2):1626-1629.
［47］Thusoo A, Sarma J S, Jain N, et al. Hive: A petabyte scale data warehouse using Hadoop［C］// IEEE 29th International Conference on Data Engineering. 2010:996-1005.
［48］梁国蓉. 一个基于Dataflow的大数据Query Engine系统的设计与实现［D］. 南京:南京大学, 2015.
［49］Shute J, Vingralek R, Samwel B, et al. F1: A distributed SQL database that scales［J］. Proceedings of the VLDB Endowment, 2013,6(11):1068-1079.
［50］Cloudera, Inc.. Apache Impala (incubating) ［EB/OL］http://www.cloudera.com/products/apache-hadoop/impala.html, 2016-06-01.
［51］Armbrust M, Xin R S, Lian C, et al. Spark SQL: Relational data processing in Spark［C］// ACM SIGMOD International Conference on Management of Data. 2015:1383-1394.

[1]	周明升, 张雯. 一种面向多源数据的智慧园区管理平台[J]. 计算机与现代化, 2023, 0(05): 68-74.
[2]	王哲, 王玉玫, 吴亚非, 臧义华. 基于RDMA的分布式键值存储系统性能优化[J]. 计算机与现代化, 2023, 0(02): 24-27.
[3]	邱金水, 庄会富, 金涛. 面向海量植物图像的智能检索系统设计[J]. 计算机与现代化, 2022, 0(10): 62-67.
[4]	单珂, 张一鸣, 刘瑞霞, . 面向中原城市群的科技服务资源池研究与设计[J]. 计算机与现代化, 2022, 0(07): 91-96.
[5]	黄安琪, 苗放, 杨文晖, 倪雅婷, 蒋媛. 基于数据架构的结构化数据注册引擎设计[J]. 计算机与现代化, 2022, 0(05): 82-89.
[6]	徐惠, 铁治欣, 舒莹. 基于特征匹配的快速降维排序搜索方法[J]. 计算机与现代化, 2022, 0(02): 85-91.
[7]	曹禹, 李晓辉, 刘忠麟, 贾贺, 费志伟. 云环境大数据工作流编排管理系统研究综述[J]. 计算机与现代化, 2022, 0(01): 41-53.
[8]	张小芳, 冯慧芳. 基于轨迹大数据的动态最优路径规划[J]. 计算机与现代化, 2021, 0(11): 82-88.
[9]	李明, 陈积富, 易小荣, 刘书铭. 基于JFinal框架的洞庭湖环境监测系统[J]. 计算机与现代化, 2021, 0(10): 41-48.
[10]	刘铮, 吴柯桦, 叶春晓. 增强安全的云存储数据访问控制方案[J]. 计算机与现代化, 2021, 0(10): 119-126.
[11]	魏云东. 基于大数据技术的人才智能推荐方法[J]. 计算机与现代化, 2021, 0(07): 60-64.
[12]	雷鸣, 姜罕盛, 武国良, 赵玉娟, 梁健. 基于HBase的大数据架构下负载平衡技术[J]. 计算机与现代化, 2021, 0(06): 91-95.
[13]	刘锋, 邹臣嵩, 崔炜. 大数据环境下基于K中心点优化算法的Web服务组合[J]. 计算机与现代化, 2020, 0(12): 20-24.
[14]	潘卫军, 刘皓晨, 王润东, 胡博文. 基于ANN的改进Spark系统在空管大数据处理中的应用[J]. 计算机与现代化, 2020, 0(12): 78-82.
[15]	雷鸣, 赵玉娟, 姜罕盛, 武国良, 梁健. 基于分布式技术的气象系统数据服务平台构建[J]. 计算机与现代化, 2020, 0(11): 56-59.