XUnified: A Framework for Guiding Optimal Use of GPU Unified Memory

Abstract

Unified Memory is a single memory address space that is accessible by any processor (GPUs or CPUs) in a system. NVIDIA&#x2019;s unified memory creates a pool of managed memory on top of physically separated CPU and GPU memories. NVIDIA&#x2019;s unified memory automatically migrates page-level data on-demand, so programmers can quickly develop CUDA codes on heterogeneous machines. However, it is extremely difficult for programmers to decide when and how to efficiently use NVIDIA&#x2019;s unified memory because (1) users are usually unaware of which unified memory hint (e.g., ReadMostly, PreferredLocation, AccessedBy) should be used for a data object in the application, and (2) it is tedious and error-prone to do manual memory management (i.e., manual code modifications) for various applications with difference data objects or inputs. We present XUnified, an <i>advice controller</i> which combines the offline training with the online adaptation to guide the optimal use of unified memory and discrete memory for various applications at runtime. The offline phase uses profiler-generated metrics to train a machine learning model, which is used to predict optimal memory advice choice and it then applies this advice to applications at runtime. We evaluate XUnified on NVIDIA Volta GPUs with a set of heterogeneous computing benchmarks. Results show that it achieves 94.0% prediction accuracy in correctly identifying the optimal memory advice choice with a maximal 34.3% reduction in kernel execution time.