Overview on Big Data Query

doi:10.3969/j.issn.1006-2475.2017.04.017

Abstract

Abstract: Big data has the characteristics such as summed up to volume, variety, velocity and veracity while traditional data doesnt have. As traditional technology doesnt meet the big data query, big data query technology has arose at the historic moment and was maturing rapidly. From the perspective of big data query, this paper analyzed the big data storage technology, big data processing platform and big data query engine, etc. This paper introduced traditional relational database, NoSQL, NewSQL and their application on big data query processing. Also, the current popular big data processing platforms and the big data query engines running on them were introduced, and an overview about their advantages and disadvantages was made.


Key words: big data, query, storage, processing platform, query engine

CLC Number:

TP301

ZHANG Yu-jie, YU Shuang-yuan. Overview on Big Data Query[J]. Computer and Modernization, 2017, 0(4): 82-88.

References

［1］徐保民. 云计算解密:技术原理及应用实践［M］. 北京:电子工业出版社, 2014.
［2］李雷. 大数据环境下数据存储与查询的研究［D］. 哈尔滨:哈尔滨工业大学, 2014.
［3］杜晓东. 大数据环境下基于Hbase的分布式查询优化研究［J］. 计算机光盘软件与应用, 2014(8):22-24.
［4］杨晶,刘天时,马刚. 分布式数据库数据分片与分配［J］. 现代电子技术, 2006,29(18):119-121.
［5］丁原,刘玉树,朱天焕. 多节点集群服务器系统共享磁盘私有网的研究［J］. 北京理工大学学报, 2001,21(1):62-64.
［6］张旭中. 分布式数据库查询优化技术［D］. 成都:电子科技大学, 2003.
［7］阮梦黎. 大数据挑战下的NoSQL系统研究［J］. 聊城大学学报(自然科学版), 2015,28(1):88-93.
［8］张滨,陈吉荣,乐嘉锦. 大数据管理技术研究综述［J］. 计算机应用与软件, 2014,31(11):1-5.
［9］ Pavlo A, Paulson E, Rasin A, et al. A comparison of approaches to large-scale data analysis［C］// ACM SIGMOD International Conference on Management of Data. 2009:165-178.
［10］Dean J, Ghemawat S. MapReduce: A flexible data processing tool［J］. Communications of the ACM, 2010,53(1):72-77.
［11］Stonebraker M, Abadi D, DeWitt D J, et al. MapReduce and parallel DBMSs: Friends or foes?［J］. Communications of the ACM, 2010,53(1):64-71.
［12］Olson M A, Bostic K, Seltzer M. Berkeley DB［C］// Freenix Track: 1999 Usenix Technical Conference. 1999:183-191.
［13］百度百科. Cassandra（开源分布式NoSQL数据库系统）［EB/OL］. http://baike.baidu.com/link?url=rgH8lTQ H4KpAWbVvCkh6oxtoUA6UoGlQDdVRhOuW3NzblbdZZW Yb4KI2bz8aaorZ63MD-V9J6Yl4_b7jzHrojEuSkGq7DTY84 qDAC2e5z-K, 2016-06-14.
［14］Neo Technology Inc. What Is Neo4j?［EB/OL］. https://neo4j.com/, 2016-06-14.
［15］百度百科. HyperGraphDB ［EB/OL］. http://baike.baidu.com/view/9329210.htm, 2016-08-14.
［16］Gilbert S, Lynch N. Brewers conjecture and the feasibility of consistent, available, partition-tolerant Web services［J］. ACM SIGACT News, 2002,33(2):51-59.
［17］Brewer E. CAP twelve years later: How the “Rules” have changed［J］. Computer, 2012,45(2):23-29.
［18］〖KG-*6〗Pavlo A, Aslett M. Whats really new with NewSQL?［J］. ACM SIGMOD Record, 2016,45(2):45-55.
［19］Corbett J C, Dean J, Epstein M, et al. Spanner: Googles globally-distributed database［C］// 10th Usenix Conference on Operating Systems Design and Implementation. 2012:251-264.
［20］VoltDB Inc.. What Is VoltDB? ［EB/OL］. https://www.voltdb.com/overview, 2013-09-29.
［21］Clustrix Inc.. A New Approach: Clustrix Sierra Database Engine［EB/OL］. http://www.clustrix.com/wp-content/uploads/2013/10/Clustrix_A-New-Approach_WhitePaper.pdf, 2013-09-29.
［22］NuoDB Greenbook Publication. NuoDB Emergent Architecture［EB/OL］. http://go.nuodb.com/rs/nuodb/images/Greenbook_Final.pdf, 2013-09-29.
［23］Grolinger K, Higashino W A, Tiwari A, et al. Data management in cloud environments: NoSQL and NewSQL data stores［J］. Journal of Cloud Computing: Advances Systems & Applications, 2013,2(1):Article No. 49.
［24］Ong K W, Papakonstantinou Y, Vernoux R. The SQL+〖KG-*3〗+ Semi-structured Data Model and Query Language: A Capabilities Survey of SQL-on-Hadoop, NoSQL and NewSQL 〖JP2〗Databases［EB/OL］. https://arxiv.org/abs/1405.3631v1, 2014-03-14.
［25］Niazi S, Ismail M, Grohsschmiedt S, et al. HopsFS: Scaling Hierarchical File System Metadata Using NewSQL Databases［EB/OL］. https://arxiv.org/abs/1606.01588, 2016-06-06.
［26］Yuan Liyan, Wu Lengdong, You Jiahuai, et al. A demonstration of Rubato DB: A highly scalable newSQL database system for OLTP and big data applications［C］// Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. 2015:907-912.
［27］Doshi K A, Zhong Tao, Lu Zhongyan, et al. Blending SQL and newSQL approaches: Reference architectures for enterprise big data challenges［C］// Proceedings of the 2013 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. 2013:163-170.
［28］Kepner J, Gadepally V, Hutchison D, et al. Associative Array Model of SQL, NoSQL, and NewSQL Databases［EB/OL］. https://arxiv.org/abs/1606.05797, 2016-06-18.
［29］Moniruzzaman A B M. NewSQL: Towards next-generation scalable RDBMS for Online Transaction Processing(OLTP) for big data management［J］. Inernational Journal of Database Theory and Application, 2014,7(6):121-130.
［30］黄宜华,苗凯翔. 深入理解大数据［M］. 北京:机械工业出版社, 2014.
［31］Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters［C］// Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation. 2004:107-113.
［32］Ghemawat S, Gobioff H, Leung S T. The Google file system［C］// Proceedings of the 19th ACM Symposium on Operating Systems Principles. 2003:29-43
［33］Chang F, Dean J, Ghemawat S, er al. Bigtable: A distributed storage system for structured data［C］// Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation. 2006:205-218.
［34］White T. Hadoop权威指南［M］. 华东师范大学数据科学与工程学院译. 北京:清华大学出版社, 2015.
［35］DeWitt D J, Stonebraker M. MapReduce: A Major Step Backwards［EB/OL］. http://homes.cs.washington.edu/~billhowe/mapreduce_a_major_step_backwards.html, 2008-01-17.
［36］谌超,强保华,石龙. 基于Hadoop MapReduce的大规模数据索引构建与集群性能分析［J］. 桂林电子科技大学学报, 2012,32(4):307-312.
［37］Murthy A C. The Next Generation of Apache Hadoop MapReduce［EB/OL］. http://www.360doc.com/content/11/0321/22/2459_103337406.shtml, 2011-03-16.
［38］Vavilapalli V K, Murthy A C, Douglas C, et al. Apache Hadoop YARN: Yet another resource negotiator［C］// Proceedings of the 4th Annual Symposium on Cloud Computing. 2013:Article No. 5.
［39］JIRA. Browse Projects［EB/OL］. https://issues.apache.org/jira/secure/BrowseProjects.jspa#10292, 2016-06-01.
［40］数盟社区. 呼之欲出！比Spark快10倍的Hadoop3.0有哪些实用新特性？［EB/OL］. http://toutiao.com/i6290843711718818306/, 2016-05-31
［41］夏俊鸾,刘旭辉,邵赛赛,等. Spark大数据处理技术［M］. 北京:电子工业出版社, 2015.
［42］Zaharia M, Chowdhury M, Das T, et al. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing［C］// Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. 2012:2.
［43］Zaharia M, Chowdhury M, Franklin M J, et al. Spark: Cluster computing with working sets［C］// Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. 2010:10.
［44］Zaharia M. An Architecture for Fast and General Data Processing on Large Clusters［M］. Association for Computing Machinery and Morgan & Claypool, 2016.
［45］于俊,向海,代其锋,等. Spark核心技术与高级应用［M］. 北京:机械工业出版社, 2016.
［46］Thusoo A, Sarma J S, Jain N, et al. Hive: A warehousing solution over a Map-Reduce framework［J］. Proceedings of the VLDB Endowment, 2011,2(2):1626-1629.
［47］Thusoo A, Sarma J S, Jain N, et al. Hive: A petabyte scale data warehouse using Hadoop［C］// IEEE 29th International Conference on Data Engineering. 2010:996-1005.
［48］梁国蓉. 一个基于Dataflow的大数据Query Engine系统的设计与实现［D］. 南京:南京大学, 2015.
［49］Shute J, Vingralek R, Samwel B, et al. F1: A distributed SQL database that scales［J］. Proceedings of the VLDB Endowment, 2013,6(11):1068-1079.
［50］Cloudera, Inc.. Apache Impala (incubating) ［EB/OL］http://www.cloudera.com/products/apache-hadoop/impala.html, 2016-06-01.
［51］Armbrust M, Xin R S, Lian C, et al. Spark SQL: Relational data processing in Spark［C］// ACM SIGMOD International Conference on Management of Data. 2015:1383-1394.

[1]	LI Junxiao1, ZHANG Xiaolin1, SHI Jing2. Blockchain-enhanced Vehicle Edge Computing Networks Security Data Storage and Share [J]. Computer and Modernization, 2024, 0(07): 69-75.
[2]	QIU Ling1, 2, SONG Zhi1, 2, LYU Shuang1, 2, YANG Xue1, 2. Application of Data Synchronization Technology in External Services of Meteorological Big Data Cloud Platform [J]. Computer and Modernization, 2024, 0(07): 76-81.
[3]	YU Chun-lei1, 2, LIU Du-jin1, ZHU Hua-wei1, YANG Jia-rong3. Fractional Repetition Codes Based on Petersen Graphs [J]. Computer and Modernization, 2024, 0(03): 122-126.
[4]	ZHANG Jun, SU Wen-hao. Optimization Method of Hadoop File Archiving Based on LZO [J]. Computer and Modernization, 2023, 0(06): 1-6.
[5]	ZHOU Ming-sheng, ZHANG Wen. A Smart Park Management Platform for Multi-source Data [J]. Computer and Modernization, 2023, 0(05): 68-74.
[6]	QIU Jin-shui, ZHUANG Hui-fu, JIN Tao. Design of Intelligent Retrieval System for Massive Plant Images [J]. Computer and Modernization, 2022, 0(10): 62-67.
[7]	SHAN Ke, ZHANG Yi-ming, LIU Rui-xia, . Research and Design of Science and Technology Service Resource Pool Oriented to Central Plains Urban Agglomeration [J]. Computer and Modernization, 2022, 0(07): 91-96.
[8]	HUANG AN-qi, MIAO Fang, YANG Wen-hui, NI Ya-ting, JIANG Yuan. Design of Structured Data Registration Engine Based on Data Architecture [J]. Computer and Modernization, 2022, 0(05): 82-89.
[9]	XU Hui, TIE Zhi-xin, SHU Ying. Fast Dimensionality Reduction Sorting Search Method Based on Feature Matching [J]. Computer and Modernization, 2022, 0(02): 85-91.
[10]	CAO Yu, LI Xiao-hui, LIU Zhong-lin, JIA He, FEI Zhi-wei. Review of Big Data Workflow Orchestration and Management System in Cloud Environment [J]. Computer and Modernization, 2022, 0(01): 41-53.
[11]	ZHANG Xiao-fang, FENG Hui-fang. Dynamic Optimal Path Planning Based on Trajectory Big Data [J]. Computer and Modernization, 2021, 0(11): 82-88.
[12]	LI Ming, CHEN Ji-fu, YI Xiao-rong, LIU Shu-ming. An Environment Monitoring System for Dongting Lake Based on JFinal Framework [J]. Computer and Modernization, 2021, 0(10): 41-48.
[13]	LIU Zheng, WU Ke-hua, YE Chun-xiao. Security-enhanced Data Access Control for Multi-authority Cloud Storage [J]. Computer and Modernization, 2021, 0(10): 119-126.
[14]	WEI Yun-dong. Intelligent Talent Recommendation Method Based on Big Data Technology [J]. Computer and Modernization, 2021, 0(07): 60-64.
[15]	LEI Ming, JIANG Han-sheng, WU Guo-liang, ZHAO Yu-juan, LIANG Jian. Load Balancing Technology Under Big Data Architecture Based on HBase [J]. Computer and Modernization, 2021, 0(06): 91-95.