基于HDF5格式的FAST望远镜数据重分布策略

doi:10.3969/j.issn.1006-2475.2019.06.010

计算机与现代化

基于HDF5格式的FAST望远镜数据重分布策略

(1.贵州大学计算机科学与技术学院,贵州贵阳550025;2.中国科学院国家天文台,北京100101)

收稿日期:2019-03-18 出版日期:2019-06-14 发布日期:2019-06-14
作者简介:钟灵(1993-),男,贵州贵阳人,硕士研究生,研究方向:数据存储,并行计算,知识图谱,E-mail: 764940915@qq.com; 李晖(1982-),男,湖南衡阳人,教授,博士,研究方向:大规模数据管理与分析,E-mail: cse.HuiLi@gzu.edu.cn; 朱明(1966-),男,广西玉林人,研究员,博士,研究方向:射电天文学,E-mail: mz@nao.cas.cn。
基金资助:
国家自然科学基金资助项目(U1531246)

Data Redistribution Strategy of FAST Telescope Based on HDF5 Format

(1. College of Computer Science and Technology, Guizhou University, Guiyang 550025, China;
2. National Astronomical Observatories, Chinese Academy of Sciences, Beijing 100101, China)

Received:2019-03-18 Online:2019-06-14 Published:2019-06-14

摘要/Abstract

摘要： 目前，世界上最大的单口径射电望远镜FAST (Five-hundred-meter Aperture Spherical radio Telescope)处于调试期，采集的数据面临着加载效率上的瓶颈，给后续的数据处理效率带来了挑战。本文结合科学数据存储格式HDF5（Hierarchical Data Format Release 5）和数据降维，提出一种适合大多数FAST望远镜数据处理场景的存储优化方法。通过优化，硬盘中的数据从二进制表格模型转换为按类型分布的多个数据集。实验结果表明本文提出的方法能够显著提高FAST望远镜的数据加载效率。

关键词: FAST望远镜, HDF5, 数据重分布, 数据结构

Abstract: At present, FAST(Five-hundred-meter Aperture Spherical radio Telescope), the world’s largest single-caliber radio telescope, is in the period of pre-run. The collected data faces bottlenecks in loading efficiency, which brings challenges to subsequent efficiencies of data processing. Combined with scientific data storage format of HDF5（Hierarchical Data Format Release 5） and data reduction, this paper proposes a storage optimization method suitable for most data processing scenarios of FAST telescope. Through optimization, data in the hard disk is converted from a binary table model to multiple data sets distributed by type. The experimental results show that the proposed method can significantly improve the data loading efficiency of FAST telescope.

Key words: FAST telescope, HDF5, data redistribution, data structure

中图分类号:

TP274

钟灵1,2,李晖1,朱明2. 基于HDF5格式的FAST望远镜数据重分布策略[J]. 计算机与现代化, doi: 10.3969/j.issn.1006-2475.2019.06.010.

ZHONG Ling1,2, LI Hui1, ZHU Ming2. Data Redistribution Strategy of FAST Telescope Based on HDF5 Format[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2019.06.010.

参考文献

［1］ COLLETTE A. Python and HDF5: Unlocking Scientific Data［M］. Sebastopol: O’Reilly Media, 2013:3-10.
［2］ FOLK M, HEBER G, KOZIOL Q, et al. An overview of the HDF5 technology suite and its applications［C］// Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases. 2011:36-47.
［3］ BYNA S, CHEN Y, SUN X H, et al. Parallel I/O prefetching using MPI file caching and I/O signatures［C］// Proceedings of 2008 ACM/IEEE Conference on Supercomputing. 2008:1-12.
［4］ GOSINK L, SHALF J, STOCKINGER K, et al. HDF5-Fastquery: Accelerating complex queries on HDF datasets using fast bitmap indices［C］// Proceedings of the 18th International Conference on Scientific and Statistical Database Management. 2006:149-158.
［5］ JENNESS T, BERRY D S, CURRIE M J, et al. Learning from 25 years of the extensible N-dimensional data format［J］. Astronomy and Computing, 2015,12:146-161.
［6］ CARLOD D F, GRSOY D, MARONE F, et al. Scientific data exchange: A schema for HDF5-based storage of raw and analyzed data［J］. Journal of Synchrotron Radiation, 2014,21(6):1224-1230.
［7］ GRAY J, LIU D T, NIETO-SANTISTEBAN M, et al. Scientific data management in the coming decade［J］. ACM Sigmod Record, 2005,34(4):34-41.
［8］ FANDANGO A, RIVERA W. High performance storage for big data analytics and visualization［M］// Handbook of Research on Big Data Storage and Visualization Techniques. IGI Global, 2018:254-275.
［9］ NAN R, LI D, JIN C, et al. The five-hundred-meter aperture spherical radio telescope (FAST) project［J］. International Journal of Modern Physics D, 2011,20(6):989-1024.
［10］裴鑫,李健,李时雨,等. (全可动射电望远镜专辑)基于多波束接收机的FRB实时搜寻终端设计［J］. 中国科学:物理学力学天文学, 2016,46(1),doi:SSPMA2018-00418.
［11］WELLS D C, GREISEN E W. FITS—A flexible image transport system［C］// Proceedings of the 5th Colloquium on Astrophysics. 1979:445-471.
［12］THOMAS B, JENNESS T, ECONOMOU F, et al. Significant problems in FITS limit its use in modern astronomical research［C］// Astronomical Data Analysis Software and Systems XXIII. 2014:351-354.
［13］HARTEN R H, GROSBOL P, GREISEN E W, et al. The FITS tables extension［J］. Astronomy and Astrophysics Supplement Series, 1988,73:365-372.
［14］PREUSS N, STAUDTER G, WEBER M, et al. Methods and technologies for research and metadata management in collaborative experimental research［J］. Applied Mechanics & Materials, 2018,885:170-183.
［15］PRICE D C, BARSDELL B R, GREENHILL L J. HDFITS: Porting the FITS data model to HDF5［J］. Astronomy and computing, 2015,12:212-220.
［16］JENNESS T. Reimplementing the hierarchical data system using HDF5［J］. Astronomy and Computing, 2015,12:221-228.
［17］COTTON W D, TODY D, PENCE W D. Binary table extension to FITS［J］. Astronomy and Astrophysics Supplement Series, 1995,113:159-166.
［18］SCROGGINS M, BOSCOE B. Once FITS, Always FITS? Astronomical Infrastructure in Transition［DB/OL］. ［2019-03-18］. https://arxiv.org/ftp/arxiv/papers/1809/1809.09224.pdf.
［19］ABADI D J, MADDEN S R, HACHEM N. Column-stores vs. row-stores: How different are they really?［C］// Proceedings of 2008 ACM SIGMOD International Conference on Management of Data. 2008:967-980.
［20］KRAUS J D, TIRUI M, RISNEN A V, et al. Radio Astronomy［M］. 2nd ed. Powell: Cygnus-Quasar Books, 1986:13,23-40.
［21］BOER J F, MILNER T E. Review of polarization sensitive optical coherence tomography and stokes vector determination［J］. Journal of biomedical optics, 2002,7(3):359-372.
［22］MCMASTER W H. Polarization and the stokes parameters［J］. American Journal of Physics, 1954,22(6):351-362.
［23］徐钧,韩金林. 射电源法拉第旋率观测效应的模拟研究［J］. 天文学报, 2011,52(4):332-338.

[1]	刘嘉昕, 游珍, 黄捷文, 陈家祥, 胡洪文, . 基于庐山三维场景的图算法虚拟仿真系统研究[J]. 计算机与现代化, 2020, 0(12): 90-98.
[2]	陈德华1,刘茜茜1,乐嘉锦1,潘乔1,朱立峰2. 病理镜检文本数据的结构化处理方法[J]. 计算机与现代化, 2016, 0(4): 1-6.
[3]	龚恒，何元清. 基于规则格网DEM的等高线绘制与优化[J]. 计算机与现代化, 2015, 0(5): 76-80.
[4]	王秋芬;袁东锋. 高校田径运动会日程自动编排算法研究[J]. 计算机与现代化, 2012, 198(2): 11-13.

基于HDF5格式的FAST望远镜数据重分布策略

Data Redistribution Strategy of FAST Telescope Based on HDF5 Format

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 4

编辑推荐

Metrics

本文评价