Data Acquisition and Storage System Based on Online Compression

doi:10.3969/j.issn.1006-2475.2014.07.011

Abstract

Abstract: In high-speed data acquisition and storage system, storage rate is expected higher and higher. Compared with the high performance of computing capability and network bandwidth, storage system has become the bottleneck of the whole system. This paper implements a data storage system. To achieve high integrity, this system implements a set of simple API and allows developers to use the specified compression algorithm. Finally, by the test, the system improves the data rate significantly, as well as the utilization rate of storage space, under the condition of a proper compression ratio.

Key words: data acquisition, storage, online compression

CLC Number:

TP319

FU Yuan-di, ZHANG Yan-yuan, LIN Yi. Data Acquisition and Storage System Based on Online Compression[J]. Computer and Modernization, doi: 10.3969/j.issn.1006-2475.2014.07.011.

References 15

[1]	王婕. 分布式网络数据采集关键技术研究[D]. 太原:中北大学, 2011.
[2]	童子权,贾俊. 采用FPGA的高速数据采集系统[J]. 国外电子测量技术, 2008,27(12):30-32.
[3]	王伟楠. 基于PCI/CPCI总线的高速数据采集卡的设计与实现[D]. 西安:西安电子科技大学, 2004.
[4]	王海明,胡金龙,吴丽华. 基于PCI总线的高速数据采集卡设计[J]. 计算机测量与控制, 2012,20(12):3393-3396.
[5]	吴鹏. 高速实时Flash阵列数据采集系统研究与实现[D]. 南京:南京理工大学, 2007.
[6]	安海磊. 高速数据采集存储系统设计[D]. 西安:西安电子科技大学, 2012.
[7]	李玉爽,刘东明,朱爱玲. 基于LZO算法的嵌入式高速无线数据采集系统[J]. 电子测量技术, 2011,34(7):67-70.
[8]	尹志勇,焦新泉,任勇峰. 雷达视频回波信号实时采集、压缩转发装置[J]. 计算机测量与控制, 2010,18(2):479-481.
[9]	幸羿南. 高速并行数据采集系统关键技术的研究[D]. 西安:西安电子科技大学, 2006.
[10]	Microsoft Corporation. File Compression and Decompression in NTFS[EB/OL]. http://msdn.microsoft.com/en-us/library/aa364219, 2013-12-21.
[11]	Google. FuseCompress[EB/OL]. http://code.google.com/p/fusecompress, 2013-12-21.
[12]	Abadi D J, Madden S R, Ferreira M C. Integrating compression and execution in column-oriented database systems[C]// Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data. 2006.
[13]	Poess M, Potapov D. Data compression in Oracle[C]// Proceedings of the 29th VLDB Conference. 2003.
[14]	Welton B, Kimpe D, Cope J, et al. Improving I/O forwarding throughput with data compression[C]// Proceedings of the 3rd Workshop on Interfaces and Abstractions for Scientific Data Storage. 2011:438-445.
[15]	Makatos T, Klonatos Y, Marazakis M, et al. ZBD: Using transparent compression at the block level to increase storage space efficiency[C]// Proceedings of the 2010 International Workshop on Storage Network Architecture and Parallel I/Os. 2010:61-70.