Data Redistribution Strategy of FAST Telescope Based on HDF5 Format

doi:10.3969/j.issn.1006-2475.2019.06.010

Abstract

Abstract: At present, FAST(Five-hundred-meter Aperture Spherical radio Telescope), the world’s largest single-caliber radio telescope, is in the period of pre-run. The collected data faces bottlenecks in loading efficiency, which brings challenges to subsequent efficiencies of data processing. Combined with scientific data storage format of HDF5（Hierarchical Data Format Release 5） and data reduction, this paper proposes a storage optimization method suitable for most data processing scenarios of FAST telescope. Through optimization, data in the hard disk is converted from a binary table model to multiple data sets distributed by type. The experimental results show that the proposed method can significantly improve the data loading efficiency of FAST telescope.

Key words: FAST telescope, HDF5, data redistribution, data structure

CLC Number:

TP274

ZHONG Ling1,2, LI Hui1, ZHU Ming2. Data Redistribution Strategy of FAST Telescope Based on HDF5 Format[J]. Computer and Modernization, 2019, 0(06): 60-.

References

［1］ COLLETTE A. Python and HDF5: Unlocking Scientific Data［M］. Sebastopol: O’Reilly Media, 2013:3-10.
［2］ FOLK M, HEBER G, KOZIOL Q, et al. An overview of the HDF5 technology suite and its applications［C］// Proceedings of the EDBT/ICDT 2011 Workshop on Array Databases. 2011:36-47.
［3］ BYNA S, CHEN Y, SUN X H, et al. Parallel I/O prefetching using MPI file caching and I/O signatures［C］// Proceedings of 2008 ACM/IEEE Conference on Supercomputing. 2008:1-12.
［4］ GOSINK L, SHALF J, STOCKINGER K, et al. HDF5-Fastquery: Accelerating complex queries on HDF datasets using fast bitmap indices［C］// Proceedings of the 18th International Conference on Scientific and Statistical Database Management. 2006:149-158.
［5］ JENNESS T, BERRY D S, CURRIE M J, et al. Learning from 25 years of the extensible N-dimensional data format［J］. Astronomy and Computing, 2015,12:146-161.
［6］ CARLOD D F, GRSOY D, MARONE F, et al. Scientific data exchange: A schema for HDF5-based storage of raw and analyzed data［J］. Journal of Synchrotron Radiation, 2014,21(6):1224-1230.
［7］ GRAY J, LIU D T, NIETO-SANTISTEBAN M, et al. Scientific data management in the coming decade［J］. ACM Sigmod Record, 2005,34(4):34-41.
［8］ FANDANGO A, RIVERA W. High performance storage for big data analytics and visualization［M］// Handbook of Research on Big Data Storage and Visualization Techniques. IGI Global, 2018:254-275.
［9］ NAN R, LI D, JIN C, et al. The five-hundred-meter aperture spherical radio telescope (FAST) project［J］. International Journal of Modern Physics D, 2011,20(6):989-1024.
［10］裴鑫,李健,李时雨,等. (全可动射电望远镜专辑)基于多波束接收机的FRB实时搜寻终端设计［J］. 中国科学:物理学力学天文学, 2016,46(1),doi:SSPMA2018-00418.
［11］WELLS D C, GREISEN E W. FITS—A flexible image transport system［C］// Proceedings of the 5th Colloquium on Astrophysics. 1979:445-471.
［12］THOMAS B, JENNESS T, ECONOMOU F, et al. Significant problems in FITS limit its use in modern astronomical research［C］// Astronomical Data Analysis Software and Systems XXIII. 2014:351-354.
［13］HARTEN R H, GROSBOL P, GREISEN E W, et al. The FITS tables extension［J］. Astronomy and Astrophysics Supplement Series, 1988,73:365-372.
［14］PREUSS N, STAUDTER G, WEBER M, et al. Methods and technologies for research and metadata management in collaborative experimental research［J］. Applied Mechanics & Materials, 2018,885:170-183.
［15］PRICE D C, BARSDELL B R, GREENHILL L J. HDFITS: Porting the FITS data model to HDF5［J］. Astronomy and computing, 2015,12:212-220.
［16］JENNESS T. Reimplementing the hierarchical data system using HDF5［J］. Astronomy and Computing, 2015,12:221-228.
［17］COTTON W D, TODY D, PENCE W D. Binary table extension to FITS［J］. Astronomy and Astrophysics Supplement Series, 1995,113:159-166.
［18］SCROGGINS M, BOSCOE B. Once FITS, Always FITS? Astronomical Infrastructure in Transition［DB/OL］. ［2019-03-18］. https://arxiv.org/ftp/arxiv/papers/1809/1809.09224.pdf.
［19］ABADI D J, MADDEN S R, HACHEM N. Column-stores vs. row-stores: How different are they really?［C］// Proceedings of 2008 ACM SIGMOD International Conference on Management of Data. 2008:967-980.
［20］KRAUS J D, TIRUI M, RISNEN A V, et al. Radio Astronomy［M］. 2nd ed. Powell: Cygnus-Quasar Books, 1986:13,23-40.
［21］BOER J F, MILNER T E. Review of polarization sensitive optical coherence tomography and stokes vector determination［J］. Journal of biomedical optics, 2002,7(3):359-372.
［22］MCMASTER W H. Polarization and the stokes parameters［J］. American Journal of Physics, 1954,22(6):351-362.
［23］徐钧,韩金林. 射电源法拉第旋率观测效应的模拟研究［J］. 天文学报, 2011,52(4):332-338.

[1]	ZHANG Yongliang1, 2, WANG Jiarun2. National Path Planning Algorithm Based on Embedded System [J]. Computer and Modernization, 2024, 0(10): 80-86.
[2]	LIU Jia-xin, YOU Zhen, HUANG Jie-wen, CHEN Jia-xiang, HU Hong-wen, . Graph Algorithm Virtual Simulation System Based on Lushan 3D Scene [J]. Computer and Modernization, 2020, 0(12): 90-98.
[3]	GONG Heng, HE Yuan-qing. Drawing and Optimization of Contour Lines Based on Grid DEM [J]. Computer and Modernization, 2015, 0(5): 76-80.
[4]	WANG Qiu-fen;YUAN Dong-feng. Study of Automatic Scheduling Algorithm for Track and Field Sports 〖JZ〗Meeting in Colleges and Universities [J]. Computer and Modernization, 2012, 198(2): 11-13.