Design of Hybrid Workload-oriented Distributed Meteorological Data Management System

doi:10.3969/j.issn.1006-2475.2023.12.020

Abstract

Abstract: Abstract: Meteorological data has the characteristics of large data scale and diverse data types. It has high requirements for access performance， so a high-performance distributed data management system is highly required. However， the meteorology is not a pure OLTP or a pure OLAP application， but a combination of the two， i.e.， HTAP with both a large number of data updating and highly concurrent data queries. Although distributed data management systems are rapidly evolving nowadays， their current support for HTAP is not very good. Therefore， in this paper， we design and implement a new hybrid workload-oriented distributed meteorological data management system for meteorological data， especially for the large-scale and most frequently used meteorological model data. The heterogeneity of different types of storage models， i.e.， grid-based and priority-based storage models， in this system can satisfy all the requirements of different types of complex queries efficiently for higher overall performance. This system can promote the concurrent query performance by 3.13 times under similar writing performance.

Key words: Key words: distributed system, data management, HTAP, meteorology, model data

CLC Number:

TP31

CHEN Chao, GU Qing-feng. Design of Hybrid Workload-oriented Distributed Meteorological Data Management System[J]. Computer and Modernization, 2023, 0(12): 118-122.

References

［1］陈景丽，陈鹏，鲍婷婷. 云计算环境下气象大数据服务应用分析［J］. 中国科技信息， 2019（11）:88.
［2］沈文海，周勇，唐伟，等. 浅论气象数据发展规划［J］. 中国信息化， 2020（9）:100-102.
［3］王怀乐，阮宇智. 大数据流式计算方法在气象领域的应用设想［J］. 信息系统工程， 2019（9）:91-94.
［4］沈文海. 再析气象大数据及其应用［J］. 中国信息化， 2016（1）:84-96.
［5］刘盼，高嵩，王若曈，等. 支撑气象大数据实时预报业务应用的MICAPS4系统概述［J］. 气象科技进展， 2018，8（1）:109-116.
［6］孟小峰，慈祥. 大数据管理:概念、技术与挑战［J］. 计算机研究与发展， 2013，50（1）:146-169.
［7］李学龙，龚海刚. 大数据系统综述［J］. 中国科学:信息科学， 2015，45（1）:1-44.
［8］ APPUSWAMY R， KARPATHIOTAKIS M， POROBIC D， et al. The case for heterogeneous HTAP［C］// Proceedings of the 8th Biennial Conference on Innovative Data Systems Research. 2017.
［9］ SIRIN U， DWARKADAS S， AILAMAKI A. Performance characterization of HTAP workloads［C］// Proceedings of the 2021 IEEE 37th International Conference on Data Engineering （ICDE）. 2021:1829-1834.
［10］ HARIZOPOULOS S， ABADI D J， MADDEN S， et al. OLTP through the looking glass， and what we found there［M］// Making Databases Work: The Pragmatic Wisdom of Michael Stonebraker. 2018:409-439.
［11］ PAVLO A， CURINO C， ZDONIK S. Skew-aware automatic database partitioning in shared-nothing， parallel OLTP systems［C］// Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 2012:61-72.
［12］ CHAUDHURI S， DAYAL U. An overview of data warehousing and OLAP technology［J］. ACM SIGMOD Record， 1997，26（1）:65-74.
［13］ CUZZOCREA A， BELLATRECHE L， SONG I Y. Data warehousing and OLAP over big data: Current challenges and future research directions［C］// Proceedings of the 16th International Workshop on Data Warehousing and OLAP. 2013:67-70.
［14］ LAHIRI T， CHAVAN S， COLGAN M， et al. Oracle database in-memory: A dual format in-memory database［C］// Proceedings of the 2015 IEEE 31st International Conference on Data Engineering. 2015:1253-1258.
［15］ MySQL HeatWave. Real-time analytics for MySQL database service［Z］. MySQL HeatWave， 2021.
［16］ LYU Z H， ZHANG H H， XIONG G， et al. Greenplum: A hybrid database for transactional and analytical workloads［C］// Proceedings of the 2021 International Conference on Management of Data. 2021:2530-2542.
［17］ CHANG F， DEAN J， GHEMAWAT S， et al. Bigtable: A distributed storage system for structured data［J］. ACM Transactions on Computer Systems （TOCS）， 2008，26（2）. DOI: 10.1145/1365815.1365816.
［18］ DECANDIA G， HASTORUN D， JAMPANI M， et al. Dynamo: Amazon’s highly available key-value store［J］. ACM SIGOPS Operating Systems Review， 2007，41（6）:205-220.
［19］ CORBETT J C， DEAN J， EPSTEIN M， et al. Spanner: Google’s globally distributed database［J］. ACM Transactions on Computer Systems （TOCS）， 2013，31（3）. DOI: 10.1145/2491245.
［20］ TAFT R， SHARIF I， MATEI A， et al. CockroachDB: The resilient geo-distributed SQL database［C］// Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 2020:1493-1509.
［21］ HUANG D X， LIU Q， CUI Q， et al. TiDB: A Raft-based HTAP database［J］. Proceedings of the VLDB Endowment， 2020，13（12）:3072-3084.
［22］ DONG S Y， KRYCZKA A， JIN Y Q， et al. RocksDB: Evolution of development priorities in a key-value store serving large-scale applications［J］. ACM Transactions on Storage （TOS）， 2021， 17（4）. DOI: 10.1145/3483840.
［23］ LAMPORT L. Paxos made simple［J］. ACM SIGACT News， 2001，32（4）:51-58.
［24］ MORARU I， ANDERSEN D G， KAMINSKY M. There is more consensus in Egalitarian parliaments［C］// Proceedings of the 24th ACM Symposium on Operating Systems Principles. 2013:358-372.
［25］ ONGARO D， OUSTERHOUT J. In search of an understandable consensus algorithm［C］// Proceedings of the 2014 USENIX Annual Technical Conference （Usenix ATC 14）. 2014:305-319.
［26］ JUNQUEIRA F P， REED B C， SERAFINI M. Zab: High-performance broadcast for primary-backup systems［C］// Proceedings of the 2011 IEEE/IFIP 41st International Conference on Dependable Systems & Networks （DSN）. 2011:245-256.
［27］ BOUTROS B S， DESAI B C. A two-phase commit protocol and its performance［C］// Proceedings of the 7th International Conference and Workshop on Database and Expert Systems Applications: DEXA 96. 1996:100-105.
［28］景苌弘，刘文洁，高锦涛，等. 面向分布式数据库的HTAP研究与实现［J］. 西北工业大学学报， 2021，39（2）:430-438.
［29］王若曈，黄向东，张博，等. 海量气象数据实时解析与存储系统的设计与实现［J］. 计算机工程与科学， 2015，37（11）:2045-2054.
［30］阿里云. 基于云上分布式NoSQL的海量气象数据存储和查询方案［DB/OL］. （2017-11-21）［2022-08-20］. https://developer.aliyun.com/article/259057.
［31］ MATSUNOBU Y， DONG S Y， LEE H. MyRocks: LSM-tree database storage engine serving Facebook’s social graph［J］. Proceedings of the VLDB Endowment， 2020，13（12）:3217-3230.

[1]	XU Jian-peng, ZHANG Hui, WU Qiong, WANG Hui, WANG Bing. Design and Application of Big Data Platform of Anhui Meteorological Service for Agriculture [J]. Computer and Modernization, 2020, 0(08): 105-108.
[2]	JU Shu-cun1,2, CHENG Wen-jie1,2, XU Jian-peng1,2, ZHOU Lu-yang1,2,. Data Acquisition System for Agricultural Meteorological IOT [J]. Computer and Modernization, 2018, 0(09): 105-.
[3]	ZHANG Chun, YUAN Tian-ning. Design of Multi-source and Heterogeneous Data Fusion Framework for Integrated Lifecycle Management of EMU [J]. Computer and Modernization, 2017, 0(10): 36-41.
[4]	LU A-lu. Design of Large-capacity Real-time Data Management System Based on RTX [J]. Computer and Modernization, 2013, 1(7): 235-238.
[5]	ZHANG Zhengwen;ZHANG Zhentao. A Public Meteorology Product System [J]. Computer and Modernization, 2013, 1(11): 143-146.
[6]	WANG Fang. SET Model of Distributed Massive Database [J]. Computer and Modernization, 2013, 218(10): 87-90.
[7]	HAN Chun-hua;LIANG Jian-feng;ZHANG Jun-ming;YIN Ru-guang. Design of Ocean Data Management and Sharing Platform [J]. Computer and Modernization, 2012, 203(7): 218-221.
[8]	HU Xing-cai. Design and Implementation of PC Software of Discovery Cam [J]. Computer and Modernization, 2012, 1(201): 201-203,.
[9]	JIANG Dai-you. Summary of Cloud Computing [J]. Computer and Modernization, 2012, 1(201): 71-73.
[10]	GUO Pengfei;LI Youxin;YAO Zhen;LIU Junjie. Development of Electric Vehicle Data Management System Based on C/S [J]. Computer and Modernization, 2012, 8(08): 198-200.
[11]	LIN Jinqun;FAN Bingbing. Design and Implementation of Data Management System of Cloud Platform [J]. Computer and Modernization, 2012, 8(08): 118-121.
[12]	GU Jin-feng;ZHANG Zi-ling. Research on Integration Technology Between SolidWorks and Small PDM System [J]. Computer and Modernization, 2011, 1(11): 214-4.
[13]	LU Qian;ZHANG Jian-jun;YAN Xiu-jing;ZHANG Xue-xia;SUI Xu-hong;. Hydro-ecological Data Management System Based on Component GIS Technology [J]. Computer and Modernization, 2010, 1(6): 44-0047.