Light: A Compatible, high-performance and scalable user-level network stack

Abstract

As the number of CPU cores and the speed of Ethernet NICs keep increasing on server machines, the network stack in the kernel has become the bottleneck for applications demanding very high throughput and ultra-low latency. Recently there is a trend towards moving the network stack out of the kernel. However, most kernel-bypass network stacks discard POSIX APIs that legacy applications have been built on, and the intricate work of transplanting applications brings the barrier to real-world deployment of kernel-bypass stacks. In this work, we propose Light, a novel user-level network stack, which not only gains highly scalable performance on multi-core server, but also achieves compatibility with legacy applications. For compatibility, Light realizes efficient blocking APIs in the user space, intercepts network-related APIs in a non-intrusive manner, and uses the FD space separation technique for proper API redirection. For high performance and scalability, Light adopts lock-free shared queue based inter-process communication and full connection affinity to reduce overheads of system call, cache miss, etc. Experiments demonstrate that many types of legacy applications could run on Light without modifying their source code. Compared with the latest kernel stack, Nginx on Light achieves up to 2.86× throughput and 78.2% lower tail latency (99.9th percentile) with 14 CPU cores.