关键词: 大规模数据处理, 用户态协议栈, 负载均衡, 内存池管理, CPU亲和性

Abstract: The traditional Linux kernel protocol stack can no longer meet the higher performance requirements of large-scale data processing systems for network transmission. A lot of researches have been done to move the protocols and interfaces from the kernel state to the user state, but there are few studies on kernel state optimization. Based on the analysis of the Linux kernel state file sending interface sendfile( ) processing flow and management mechanism, this paper proposes a kernel state file sending optimization method, using automatic load balancing fixed-length memory pool management, CPU affinity technologies, the kernel state interface sendfile( ) has been optimized and modified. It solves the problems of memory fragmentation, memory exhaust and CPU jitter under high load conditions, and improves data transmission performance effectively. Experimental results show that in the high concurrent and high throughput scenario, the system using this optimization method runs more stably, and the kernel state CPU occupancy rate drops by 50%.

Key words: large-scale data processing, user state protocol stack； load balance； memory pool management； CPU affinity