VProbe: Scheduling Virtual Machines on NUMA Systems

Abstract

With the development of multi-core platforms andcloud computing, Non-Uniform Memory Access (NUMA) architecturehas been dominant in cloud data centers in recentyears. However, NUMA architecture is not well supported invirtualized environments. Because of the semantic gap introducedby the virtualization layer, hypervisors know little aboutthe characteristics of applications running in virtual machines(VMs). More importantly, in order to guarantee hypervisors'applicability, load balance strategies of virtual CPU (VCPU) schedulers do not consider the memory access characteristicsof applications running in VMs, which probably introducessignificant shared resource contention and unnecessary remotememory accesses. In this paper, we propose a NUMA-aware VCPU schedulerbased on Xen, named vProbe, to improve the performanceof memory-intensive applications while maintaining the transparencyof the virtualization layer in NUMA-based servers. It collects performance monitoring units (PMU) data for eachVCPU and analyzes their memory access characteristics. Then, according to the memory access characteristics of each VCPU, it periodically reassigns all memory-intensive VCPUs to eachNUMA node evenly while preferentially allocating them to theirlocal nodes, which aims to alleviate shared resource contentionand reduce unnecessary remote memory accesses. Moreover, when a physical CPU (PCPU) becomes idle, it preferentiallysteals a VCPU from the run queues of PCPUs in the local nodeto this PCPU, which helps to maintain balanced last-level cache(LLC) contention and reduce extra remote memory accesses. Our evaluation shows that vProbe can significantly improve theperformance of memory-intensive applications (e.g., up to 45.2%performance improvement compared with the Credit scheduler) while introducing negligible overheads.