Making Stateless and Stateful Network Performance Measurements Unbiased

Abstract

The Benchmarking Working Group (BMWG) of the Internet Engineering Task Force (IETF) has defined a series of Requests for Comments (RFC) to standardize the benchmarking of network interconnect devices (e.g., bridges, routers, different IPv6 transition solutions). The paper points out that there are cases where the performance results are significantly different when a single IP address pair or multiple IP addresses are used. The cause of this phenomenon is rooted in the recent hardware and software advancements: Receive Side Scaling (RSS) makes it possible to distribute packet processing workload over multiple CPU cores. However, this may be implemented in two ways: the first way only includes the IP addresses into the hash function used to distribute the workload among the CPU cores, whereas the second one also includes the port numbers. RFC 4814 proposed an excellent solution for the second case by recommending the usage of pseudorandom port numbers during benchmarking; however, the first case was not handled properly, because no explicit recommendation was given regarding the usage of multiple IP addresses. This paper attempts to bridge this methodological gap; a practical solution is proposed for using pseudorandom IP addresses in various scenarios including the benchmarking of IPv4 and IPv6 routers and Network Address Translation from IPv6 Clients to IPv4 Servers (stateful NAT64) gateways. Its feasibility is shown by disclosing the details of its implementation in siitperf. Then the proposed solution is validated by both stateless and stateful tests. It is shown that the measurement results of the tests following the proposed solution can better characterize the true performance of the network interconnect devices that follow the first type of RSS implementation than the results of the tests using a single IP address pair.