Computer and Modernization ›› 2024, Vol. 0 ›› Issue (07): 76-81.doi: 10.3969/j.issn.1006-2475.2024.07.011

Previous Articles     Next Articles

Application of Data Synchronization Technology in External Services of Meteorological Big Data Cloud Platform

  

  1. (1. Sichuan Meteorological Observation and Data Centre, Chengdu 610072, China; 2. Heavy Rain and Drought-Flood Disasters in Plateau and Basin Key Laboratory of Sichuan Province, Chengdu 610072, China)
  • Online:2024-07-25 Published:2024-08-08

Abstract: Currently, the meteorological big data cloud platform exclusively serves meteorological Intranet users. In order to further expand the platform’s capability for external services and meet the extensive demands of meteorological industry users for the storage and sharing of massive meteorological data applications, a secure and efficient meteorological big data storage and service system is designed based on key technologies such as distributed storage and data service interfaces. This system is implemented in the Unidirectional Network Security Isolation Zone (Demilitarized Zone, DMZ). For structured meteorological data, an optimized data synchronization method based on the Database Binlog in a distributed database is researched and designed. For unstructured meteorological data, an optimized data synchronization method combining the Database Binlog and ETL mechanisms is researched and designed to achieve real-time second-level unidirectional synchronization of massive meteorological data between the meteorological intranet big data cloud platform and the DMZ area. The method proposed in this paper addresses issues such as low data volume, poor sharing service capabilities, and low security for meteorological data in industry user services. Application practices demonstrate that the data synchronization technology achieves synchronization delays within seconds,maintains good data consistency, and ensures the real-time and accurate external service of the meteorological big data cloud platform.

Key words:  , data synchronization, Binlog, ETL, meteorological big data, DMZ zone

CLC Number: