关键词: 磁盘缓存, 虚拟存储, 文件预取, 缓存替换, 小文件合并

Abstract: The Hadoop distributed file system (HDFS CD-ROM database) based on CD-ROM database meets the current requirements of large data storage in terms of unit storage cost, data security and service life, etc., but it is not suitable for storing a large number of small files and real-time data reading. To better apply HDFS CD-ROM database in more big data storage scenarios, this paper proposes a magneto-optical virtual storage system (MOVS) more suitable for big data storage, which adds disk cache between HDFS CD-ROM database and users, and merges small files in disk cache into large files suitable for HDFS CD-ROM storage through file label classification, virtual storage, small file merging and other technologies, improving the data transmission speed. The system also uses file scheduling algorithm such as file pre-fetching and cache replacement to dynamically update the files in disk cache, so as to minimize the number of HDFS CD-ROM database accesses. The results of experiment show that MOVS can greatly improve the response time and data transmission speed compared with HDFS CD-ROM database.

Key words: disk cache, virtual storage, file pre-fetching, cache replacement, small file merging