关键词: 自动站分钟数据, Kafka, HBase, Elasticsearch, 协处理器

Abstract:  With the rapid growth of temporal and spatial density of ground automatic weather station, the quantity of observed data increases exponentially, traditional relational database has insufficient ability and low performance in mass data storage and retrieval. In view of this, an automatic station minute data storage and retrieval system is designed. Quartz is used to collect automatic station minute files and then the system decodes the minute files and storage decoded data into database. The system uses HBase to build minute data storage model. In order to meet the query demands of multiple elements, the system applies Elasticsearch to build secondary index of HBase. Test results show that the average storage time for minute data is 54.6 s and the secondary index with Elasticsearch is complete and reliable and the retrieval results returned time to milliseconds. Therefore, the system can satisfy the demand for storage and query performance of automatic weather station minute data.

Key words: AWS minute data, Kafka, HBase, Elasticsearch, coprocessor