Latency Optimization of Distributed Key-value Store Based on RDMA

Abstract

Abstract: Distributed key-value store is widely used in industry， and methods to improve the performance of data transfer in distributed key-value store has drawn great interests in the study of distributed systems and parallel computing. RDMA （Remote Memory Direct Access） is a popular network technology in high-performance computing. Making use of its low latency and high bandwidth features can lead to enormous improvement of the performance of distributed key-value store. MPI offers routines to perform remote memory access （RMA）， also known as one-sided communication， which can benefit from RDMA technology. In this paper， we propose leverageing MPI one-sided communication refactors key-value operations and redesigns the communication model according to the features of one-sided communication. To solve the data inconsistency problem brought by RMA， we modify the storage structure of key-value pair and schedule the communication strategy. We evaluate the improvement by comparing the latency of RMA based key-value operations and TCP/IP based key-value operations when transferring key-value pairs of different lengths and get positive result.

Key words: distributed key-value store, RDMA, one-sided communication

WANG Zhe, WANG Yu-mei, WU Ya-fei, ZANG Yi-hua. Latency Optimization of Distributed Key-value Store Based on RDMA[J]. Computer and Modernization, 2023, 0(02): 24-27.

References

［1］ YANG J， YUE Y， RASHMI K V. A large-scale analysis of hundreds of in-memory key-value cache clusters at Twitter［J］. ACM Transactions on Storage， 2021，17（3）:1-35.
［2］ ROMANOW A， BAILEY S. An overview of RDMA over IP［C/OL］// Proceedings of the 1st International Workshop on Protocols for Fast Long-Distance Networks， 2003. http://datatag.web.cern.ch/pfldnet2003/paper romanow.pdf.
［3］涂晓军，孙权，蔡立志. RDMA技术在数据中心中的应用研究［J］. 计算机应用与软件， 2021，38（3）:22-25.
［4］ KALIA A， KAMINSKY M， ANDERSEN D G. Using RDMA efficiently for key-value services［C］// Proceedings of the 2014 ACM Conference on SIGCOMM. 2014:295-306.
［5］ ARLITT M F， WILLIAMSON C L. Internet Web servers: Workload characterization and performance implications［J］. IEEE/ACM Transactions on Networking， 1997，5（5）:631-645.
［6］ NETFIX.Memcached［CP/OL］. ［2022-05-21］. https://memcached.org/.
［7］ REDIS LID. Redis［CP/OL］. ［2022-05-21］. https://redis.io/.
［8］李首扬，杨仝，代亚非，等. Memcached的分析与改进［J］. 网络新媒体技术， 2016（6）:38-42.
［9］ FITZPATRICK B. Distributed caching with Memcached［J］. Linux Journal， 2004（124）:72-74.
［10］刘亮，徐步东，谭艳艳. 基于Memcached内存对象缓存技术应用研究［J］. 计算机技术与发展， 2015，25（11）:204-208.
［11］马豫星. Redis数据库特性分析［J］. 物联网技术， 2015，5（3）:105-106.
［12］王绍东. 基于Redis Cluster的分布式内存数据库研究与应用［D］. 广州:华南理工大学， 2016.
［13］ FOONG A P， HUFF T R， HUM H H， et al. TCP performance re-visited［C］// Proceedings of the 2003 IEEE International Symposium on Performance Analysis of Systems and Software. 2003:70-79.
［14］ PISANI F， PÉREZ D H C， NEUFELD N. High-speed zero-copy data transfer for DAQ applications［J］. Journal of Physics Conference Series， 2015，608（1）. DOI:10.1088/1742-6596/608/1/012028.
［15］ CHAPMAN B， CURTIS T， POPHALE S， et al. Introducing OpenSHMEM: SHMEM for the PGAS community［C］// Proceedings of the 4th Conference on Partitioned Global Address Space Programming Model. 2010:1-3.
［16］ Maynard C. Comparing one-sided communication with MPI， UPC and SHMEM［C］// Cray User Group 2012 Conference. 2012.
［17］ DINAN J， BALAJI P， BUNTINAS D， et al. An implementation and evaluation of the MPI 3.0 one-sided communication interface［J］. Concurrency and Computation Practice and Experience， 2016，28（17）. DOI:10.1002/cpe.3758.
［18］周寰，郑启龙，汪睿. 基于单边通信的MPI广播算法［J］. 中国科学技术大学学报， 2011，41（4）:334-340.
［19］ HOEFLER T， DINAN J， THAKUR R， et al. Remote memory access programming in MPI-3［J］. ACM Transactions on Parallel Computing， 2015，2（2）:1-26.
［20］ PANDA D K， SUBRAMONI H， CHU C， et al. The MVAPICH project: Transforming research into high-performance MPI library for HPC community［J］. Journal of Computational Science， 2021，52. DOI:10.1016/j.jocs.2020.101208.
［21］ OPEN MPI. OpenMPI［CP/OL］. ［2022-05-21］. https://open-mpi.org.
［22］沈佳杰，卢修文，向望，等. 分布式存储系统读写一致性算法性能优化研究综述［J］. 计算机工程与科学， 2022，44（4）:571-583.
［23］ DRAGOJEVIĆ A， NARAYANAN D， HODSON O， et al. FaRM: Fast remote memory［C］// Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation. 2014:401-414.
［24］ MITCHELL C， GENG Y， LI J. Using one-sided RDMA reads to build a fast， CPU-efficient key-value store［C］// Proceedings of the 2013 USENIX Conference on Annual Technical Conference. 2013:103-114.
［25］ JOSE J， SUBRAMONI H， LUO M， et al. Memcached design on high performance RDMA capable interconnects［C］// Proceedings of the 2011 International Conference on Parallel Processing. 2011:743-752.
［26］ FAN B， ANDERSEN D G， KAMINSKY M. MemC3: Compact and concurrent MemCache with dumber caching and smarter hashing［C］// Proceedings of the 10th USENIX conference on Networked Systems Design and Implementation. 2013:371-384.
［27］ WANG Y D， ZHANG L， TAN J， et al. HydraDB: A resilient RDMA-driven key-value middleware for in-memory cluster computing［C］// Proceedings of the International Conference for High Performance Computing， Networking， Storage and Analysis. 2015:1-11.