Unified Memory Research Articles

Unified Holistic Memory Management Supporting Multiple Big Data Processing Frameworks over Hybrid Memories

To process real-world datasets, modern data-parallel systems often require extremely large amounts of memory, which are both costly and energy inefficient. Emerging non-volatile memory (NVM) technologies offer high capacity compared to DRAM and low energy compared to SSDs. Hence, NVMs have the potential to fundamentally change the dichotomy between DRAM and durable storage in Big Data processing. However, most Big Data applications are written in managed languages and executed on top of a managed runtime that already performs various dimensions of memory management. Supporting hybrid physical memories adds a new dimension, creating unique challenges in data replacement. This article proposes Panthera, a semantics-aware, fully automated memory management technique for Big Data processing over hybrid memories. Panthera analyzes user programs on a Big Data system to infer their coarse-grained access patterns, which are then passed to the Panthera runtime for efficient data placement and migration. For Big Data applications, the coarse-grained data division information is accurate enough to guide the GC for data layout, which hardly incurs overhead in data monitoring and moving. We implemented Panthera in OpenJDK and Apache Spark. Based on Big Data applications’ memory access pattern, we also implemented a new profiling-guided optimization strategy, which is transparent to applications. With this optimization, our extensive evaluation demonstrates that Panthera reduces energy by 32–53% at less than 1% time overhead on average. To show Panthera’s applicability, we extend it to QuickCached, a pure Java implementation of Memcached. Our evaluation results show that Panthera reduces energy by 28.7% at 5.2% time overhead on average.

Efficient Use of GPU Memory for Large-Scale Deep Learning Model Training

To achieve high accuracy when performing deep learning, it is necessary to use a large-scale training model. However, due to the limitations of GPU memory, it is difficult to train large-scale training models within a single GPU. NVIDIA introduced a technology called CUDA Unified Memory with CUDA 6 to overcome the limitations of GPU memory by virtually combining GPU memory and CPU memory. In addition, in CUDA 8, memory advise options are introduced to efficiently utilize CUDA Unified Memory. In this work, we propose a newly optimized scheme based on CUDA Unified Memory to efficiently use GPU memory by applying different memory advise to each data type according to access patterns in deep learning training. We apply CUDA Unified Memory technology to PyTorch to see the performance of large-scale learning models through the expanded GPU memory. We conduct comprehensive experiments on how to efficiently utilize Unified Memory by applying memory advises when performing deep learning. As a result, when the data used for deep learning are divided into three types and a memory advise is applied to the data according to the access pattern, the deep learning execution time is reduced by 9.4% compared to the default Unified Memory.

Multi-GPU systems and Unified Virtual Memory for scientific applications: The case of the NAS multi-zone parallel benchmarks

GPU-based computing systems have become a widely accepted solution for the high-performance-computing (HPC) domain. GPUs have shown highly competitive performance-per-watt ratios and can exploit an astonishing level of parallelism. However, exploiting the peak performance of such devices is a challenge, mainly due to the combination of two essential aspects of multi-GPU execution: memory allocation and work distribution. Memory allocation determines the data mapping to GPUs, and therefore conditions all work distribution schemes and communication phases in the application. Unified Virtual Memory simplifies the codification of memory allocations, but its effects on performance depend on how data is used by the devices and how the devices' driver is going to orchestrate the data transfers across the system.In this paper we present a multi-GPU and Unified Virtual Memory (UM) implementation of the NAS Multi-Zone Parallel Benchmarks which alternate communication and computation phases offering opportunities to overlap these phases. We analyse the programmability and performance effects of the introduction of the UM support.Our experience shows that the programming efforts for introducing UM are similar to those of having a memory allocation per GPU. On an evaluation environment composed of 2 x IBM Power9 8335-GTH and 4 x GPU NVIDIA V100 (Volta), our UM-based parallelization outperforms the manual memory allocation versions by 1.10x to 1.85x. However, these improvements are highly sensitive to the information forwarded to the devices' driver describing the most convenient location for specific memory regions. We analyse these improvements in terms of the relationship between the computational and communication phases of the applications.

Efficient ROS-Compliant CPU-iGPU Communication on Embedded Platforms

Many modern programmable embedded devices contain CPUs and a GPU that share the same system memory on a single die. Such a unified memory architecture (UMA) allows programmers to implement different communication models between CPU and the integrated GPU (iGPU). Although the simpler model guarantees implicit synchronization at the cost of performance, the more advanced model allows, through the zero-copy paradigm, the explicit data copying between CPU and iGPU to be eliminated with the benefit of significantly improving performance and energy savings. On the other hand, the robot operating system (ROS) has become a de-facto reference standard for developing robotic applications. It allows for application re-use and the easy integration of software blocks in complex cyber-physical systems. Although ROS compliance is strongly required for SW portability and reuse, it can lead to performance loss and elude the benefits of the zero-copy communication. In this article we present efficient techniques to implement CPU–iGPU communication by guaranteeing compliance to the ROS standard. We show how key features of each communication model are maintained and the corresponding overhead involved by the ROS compliancy.

Accelerating On-Device Learning with Layer-Wise Processor Selection Method on Unified Memory.

Recent studies have applied the superior performance of deep learning to mobile devices, and these studies have enabled the running of the deep learning model on a mobile device with limited computing power. However, there is performance degradation of the deep learning model when it is deployed in mobile devices, due to the different sensors of each device. To solve this issue, it is necessary to train a network model specific to each mobile device. Therefore, herein, we propose an acceleration method for on-device learning to mitigate the device heterogeneity. The proposed method efficiently utilizes unified memory for reducing the latency of data transfer during network model training. In addition, we propose the layer-wise processor selection method to consider the latency generated by the difference in the processor performing the forward propagation step and the backpropagation step in the same layer. The experiments were performed on an ODROID-XU4 with the ResNet-18 model, and the experimental results indicate that the proposed method reduces the latency by at most 28.4% compared to the central processing unit (CPU) and at most 21.8% compared to the graphics processing unit (GPU). Through experiments using various batch sizes to measure the average power consumption, we confirmed that device heterogeneity is alleviated by performing on-device learning using the proposed method.

An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning

Deep learning for digital pathology is hindered by the extremely high spatial resolution of whole-slide images (WSIs). Most studies have employed patch-based methods, which often require detailed annotation of image patches. This typically involves laborious free-hand contouring on WSIs. To alleviate the burden of such contouring and obtain benefits from scaling up training with numerous WSIs, we develop a method for training neural networks on entire WSIs using only slide-level diagnoses. Our method leverages the unified memory mechanism to overcome the memory constraint of compute accelerators. Experiments conducted on a data set of 9662 lung cancer WSIs reveal that the proposed method achieves areas under the receiver operating characteristic curve of 0.9594 and 0.9414 for adenocarcinoma and squamous cell carcinoma classification on the testing set, respectively. Furthermore, the method demonstrates higher classification performance than multiple-instance learning as well as strong localization results for small lesions through class activation mapping.

Grus

Today’s GPU graph processing frameworks face scalability and efficiency issues as the graph size exceeds GPU-dedicated memory limit. Although recent GPUs can over-subscribe memory with Unified Memory (UM), they incur significant overhead when handling graph-structured data. In addition, many popular processing frameworks suffer sub-optimal efficiency due to heavy atomic operations when tracking the active vertices. This article presents Grus, a novel system framework that allows GPU graph processing to stay competitive with the ever-growing graph complexity. Grus improves space efficiency through a UM trimming scheme tailored to the data access behaviors of graph workloads. It also uses a lightweight frontier structure to further reduce atomic operations. With easy-to-use interface that abstracts the above details, Grus shows up to 6.4× average speedup over the state-of-the-art in-memory GPU graph processing framework. It allows one to process large graphs of 5.5 billion edges in seconds with a single GPU.

A Cuda Fast Multipole Method with Highly Efficient M2L Far Field Evaluation

Solving an N-body problem is a computationally quite demanding task in many scientific fields ranging from astrophysics to biomolecular simulations. However, the direct solution scales with O(N2); hence, even on modern hardware, the direct calculation becomes impractical even for moderate numbers of particles, and thus, efficient yet accurate approximations are key. For biomolecular simulations, a widely used such method is Particle Mesh Ewald (PME), which scales with O(N log N). Although extremely fast on a single node, PME runs into communication bottlenecks when parallelized for large simulation systems on many nodes. The fast multipole method (FMM) offers an attractive alternative: It requires less communication and reduces the complexity to optimal O(N). The method approximates long-range interactions by grouping the particles into clusters represented as multipoles. The cluster size grows with the interaction distance according to the underlying octree structure. Hence, further separated particles require fewer interaction computations, hence less communication. Here, we present our full NVIDIA CUDA FMM implementation, which has been optimized for the electrostatic interactions described by Coulomb's law relevant to molecular dynamics simulations. We compare different parallelization approaches to the computationally limiting part of the algorithm, the Multipole-to-Local (M2L) operator, and discuss their performance bottlenecks. The first approach can be implemented with only minimal modifications to the sequential CPU implementation. It features the Unified Memory concept, which allows for a simple utilization of the existing CPU data structures. The second approach enhances the achieved performance by exploiting CUDA Dynamic Parallelism. It introduces a significant speedup, especially for a high accuracy requirement. The third parallelization approach abstracts the underlying octree with precomputed interaction lists and it exploits operator symmetries to achieve nearly optimal performance in the whole tested accuracy range.

Performance of CUDA Unified Memory in CMS Heterogeneous Pixel Reconstruction

The management of separate memory spaces of CPUs and GPUs brings an additional burden to the development of software for GPUs. To help with this, CUDA unified memory provides a single address space that can be accessed from both CPU and GPU. The automatic data transfer mechanism is based on page faults generated by the memory accesses. This mechanism has a performance cost, that can be with explicit memory prefetch requests. Various hints on the inteded usage of the memory regions can also be given to further improve the performance. The overall effect of unified memory compared to an explicit memory management can depend heavily on the application. In this paper we evaluate the performance impact of CUDA unified memory using the heterogeneous pixel reconstruction code from the CMS experiment as a realistic use case of a GPU-targeting HEP reconstruction software. We also compare the programming model using CUDA unified memory to the explicit management of separate CPU and GPU memory spaces.

Second Kings 24–25 and Jeremiah 52 as Diverging and Converging Memories of the Babylonian Conquest

The accounts of the Babylonian conquest of Jerusalem in Jeremiah 52 and 2 Kings 24:18–25:30 are virtually identical and seem to convey a unified memory of this historical event. However, a closer examination of the Hebrew and Greek texts reveals that these nearly identical accounts are the result of a longer process of textual changes. The unified memory on the surface conceals an underlying pluriformity of memories. A comparison between the account in 2 Kings 24:18–25:30 and the parallels in Jeremiah can thus serve as a case study on how the Babylonian conquest was construed as a cultural trauma in ancient Israel's collective memory.

High throughput token driven FSM based regex pattern matching for network intrusion detection system

A single core SRAM memory with a unified memory controller is proposed for regex patterns which contains complex signatures. Many existing NIDS systems are based on token stream with dedicated hardware units to accommodate byte oriented matching with moderate network throughput rate. The overall NIDS performance metrics is degraded when the number of bytes in the database increased. It will lead the demands of coordinated payload monitoring scheme in any memory based digital NIDS system. In this work, a bit based pattern matching algorithm with FSM state transition is proposed which gives both parallel processing and network assertion payload validity check during the conversion of segments into tokens. The Finite State Machine state controller is used in early misdetection to optimize overall matching time in case of segments with multiple non-trivial tokens where the regex patterns are divided into sub-patterns and the concurrent matches are halted in a parallel manner. Finally synchronization problem during hierarchical state transition and vector matching over multiple pages unique SRAM controllers are used which is driven with fully down converted sub-groups. The efficiency of proposed NIDS system over moderate intrusion database process is proved in terms of classification rate. Hence, proposed NIDS system consumes lesser memory resources and it is verified through comparison with state-of-the-art methods.

SA-JSTN: Self-Attention Joint Spatiotemporal Network for Temperature Forecasting

The rapid development of remote sensing technology has brought abundant data support for deep learning based temperature forecasting research. However, recently proposed methods usually focus on the temporal relationship among temperature observation information, whereas ignore the spatial positions of different regions. Motivated by the observation that adjacent regions usually present similar temperature trends, in this article, we consider the temperature forecasting as a spatiotemporal sequence prediction problem, and propose a new deep learning model for temperature forecasting, self-attention joint spatiotemporal network (SA-JSTN), which simultaneously captures the spatiotemporal interdependency information. The kernel component of the SA-JSTN is a newly developed spatiotemporal memory (STM) unit, which describes the temporal and spatial models via a unified memory cell. STM is constructed based on the units of the convolutional long short-term memory (ConvLSTM). Instead of using simple convolutions for spatial information extraction, in STM, we improve ConvLSTM by a self-attention module, which has significantly enhanced the global spatial information representation ability of our proposed network. Compared with other deep learning based temperature forecasting methods, SA-JSTN is able to integrate the global spatial correlation into the temperature series prediction problem, and thus present better performance especially in short-term prediction. We have conducted comparison experiments on two typical temperature datasets to validate the effectiveness of our proposed method.

Optimizing Data Pipeline Performance in Modern GPU Architectures

Optimizing data pipeline performance in modern GPU architectures is critical for achieving high computational throughput and efficient resource utilization in data-intensive applications. With the rise of deep learning, scientific simulations, and real-time analytics, GPUs have become integral in accelerating data processing tasks. However, ensuring optimal performance involves addressing several challenges, such as memory bandwidth limitations, data transfer bottlenecks between CPU and GPU, and efficient parallel execution of workloads. This research explores techniques for improving data pipeline performance by focusing on memory management, load balancing, and task scheduling. One key strategy is optimizing data movement through techniques like memory coalescing, which minimizes access latency, and overlapping data transfers with computation. Furthermore, leveraging the architectural advances in modern GPUs, such as unified memory and NVLink, can significantly reduce data transfer overhead. Task parallelism and efficient workload distribution across multiple GPU cores also play a crucial role in enhancing pipeline throughput. Additionally, the study highlights the importance of tuning GPU kernels and optimizing data preprocessing steps to ensure minimal latency and maximum throughput. By adopting advanced profiling tools and performance monitoring techniques, bottlenecks can be identified, and pipeline optimization strategies can be fine-tuned. The findings presented provide a comprehensive approach for designing and optimizing data pipelines, leading to significant performance improvements in GPU-based systems, ultimately driving the next generation of high-performance computing applications.

IMLBench: A Machine Learning Benchmark Suite for CPU-GPU Integrated Architectures

Utilizing heterogeneous accelerators, especially GPUs, to accelerate machine learning tasks has shown to be a great success in recent years. GPUs bring huge performance improvements to machine learning and greatly promote the widespread adoption of machine learning. However, the discrete CPU-GPU architecture design with high PCIe transmission overhead decreases the GPU computing benefits in machine learning training tasks. To overcome such limitations, hardware vendors release CPU-GPU integrated architectures with shared unified memory. In this article, we design a benchmark suite for machine learning training on CPU-GPU integrated architectures, called iMLBench, covering a wide range of machine learning applications and kernels. We mainly explore two features on integrated architectures: 1) zero-copy, which means that the PCIe overhead has been eliminated for machine learning tasks and 2) co-running, which means that the CPU and the GPU co-run together to process a single machine learning task. Our experimental results on iMLBench show that the integrated architecture brings an average 7.1× performance improvement over the original implementations. Specifically, the zero-copy design brings 4.65× performance improvement, and co-running brings 1.78× improvement. Moreover, integrated architectures exhibit promising results from both performance-per-dollar and energy perspectives, achieving 6.50× performance-price ratio while 4.06× energy efficiency over discrete GPUs. The benchmark is open-sourced at https://github.com/ChenyangZhang-cs/iMLBench.

Efficient Buffer Overflow Detection on GPU

Rich thread-level parallelism of GPU has motivated co-running GPU kernels on a single GPU. However, when GPU kernels co-run, it is possible that one kernel can leverage buffer overflow to attack another kernel running on the same GPU. There is very limited work aiming to detect buffer overflow for GPU. Existing work has either large performance overhead or limited capability in detecting buffer overflow. In this article, we introduce GMODx, a runtime software system that can detect GPU buffer overflow. GMODx performs always-on monitoring on allocated memory based on a canary-based design. First , for the fine-grained memory management, GMODx introduces a set of byte arrays to store buffer information for overflow detection. Techniques, such as lock-free accesses to the byte arrays, delayed memory free, efficient memory reallocation, and garbage collection for the byte arrays, are proposed to achieve high performance. Second , for the coarse-grained memory management, GMODx utilizes unified memory to delegate the always-on monitoring to the CPU. To reduce performance overhead, we propose several techniques, including customized list data structure and specific optimizations against the unified memory. For micro-benchmarking, our experiments show that GMODx is capable of detecting buffer overflow for the fine-grained memory management without performance loss, and that it incurs small runtime overhead (4.2 percent on average and up to 9.7 percent) for the coarse-grained memory management. For real workloads, we deploy GMODx on the TensorFlow framework, it only causes 0.8 percent overhead on average (up to 1.8 percent).

A Data-Centric Directive-Based Framework to Accelerate Out-of-Core Stencil Computation on a GPU

Graphics processing units (GPUs) are highly efficient architectures for parallel stencil code; however, the small device (i.e., GPU) memory capacity (several tens of GBs) necessitates the use of out-of-core computation to process excess data. Great programming effort is needed to manually implement efficient out-of-core stencil code. To relieve such programming burdens, directive-based frameworks emerged, such as the pipelined accelerator (PACC); however, they usually lack specific optimizations to reduce data transfer. In this paper, we extend PACC with two data-centric optimizations to address data transfer problems. The first is a direct-mapping scheme that eliminates host (i.e., CPU) buffers, which intermediate between the original data and device buffers. The second is a region-sharing scheme that significantly reduces host-to-device data transfer. The extended PACC was applied to an acoustic wave propagator, automatically extending the length of original serial code 2.3-fold to obtain the out-of-core code. Experimental results revealed that on a Tesla V100 GPU, the generated code ran 41.0, 22.1, and 3.6 times as fast as implementations based on Open Multi-Processing (OpenMP), Unified Memory, and the previous PACC, respectively. The generated code also demonstrated usefulness with small datasets that fit in the device capacity, running 1.3 times as fast as an in-core implementation.

Jittor: a novel deep learning framework with meta-operators and unified graph execution

This paper introduces Jittor, a fully just-in-time (JIT) compiled deep learning framework. With JIT compilation, we can achieve higher performance while making systems highly customizable. Jittor provides classes of Numpy-like operators, which we call meta-operators. A deep learning model built upon these meta-operators is compiled into high-performance CPU or GPU code in real-time. To manage metaoperators, Jittor uses a highly optimized way of executing computation graphs, which we call unified graph execution. This approach is as easy to use as dynamic graph execution yet has the efficiency of static graph execution. It also provides other improvements, including operator fusion, cross iteration fusion, and unified memory.

A CUDA fast multipole method with highly efficient M2L far field evaluation

Solving an N-body problem, electrostatic or gravitational, is a crucial task and the main computational bottleneck in many scientific applications. Its direct solution is an ubiquitous showcase example for the compute power of graphics processing units (GPUs). However, the naïve pairwise summation has [Formula: see text] computational complexity. The fast multipole method (FMM) can reduce runtime and complexity to [Formula: see text] for any specified precision. Here, we present a CUDA-accelerated, C++ FMM implementation for multi particle systems with [Formula: see text] potential that are found, e.g. in biomolecular simulations. The algorithm involves several operators to exchange information in an octree data structure. We focus on the Multipole-to-Local (M2L) operator, as its runtime is limiting for the overall performance. We propose, implement and benchmark three different M2L parallelization approaches. Approach (1) utilizes Unified Memory to minimize programming and porting efforts. It achieves decent speedups for only little implementation work. Approach (2) employs CUDA Dynamic Parallelism to significantly improve performance for high approximation accuracies. The presorted list-based approach (3) fits periodic boundary conditions particularly well. It exploits FMM operator symmetries to minimize both memory access and the number of complex multiplications. The result is a compute-bound implementation, i.e. performance is limited by arithmetic operations rather than by memory accesses. The complete CUDA parallelized FMM is incorporated within the GROMACS molecular dynamics package as an alternative Coulomb solver.

A thread‐block‐wise computational framework for large‐scale hierarchical continuum‐discrete modeling of granular media

AbstractThis article presents a novel, scalable parallel computing framework for large‐scale and multiscale simulations of granular media. Key to the new framework is an innovative thread‐block‐wise representative volume element (RVE) parallelism, inspired by the resemblance between a typical multiscale computational hierarchy and the hierarchical thread structure of graphics processing units (GPUs). To solve a hierarchical multiscale problem, all computation in an RVE is assigned a single block of threads so that the RVE runs entirely on a GPU to avoid frequent data exchange with the host CPU. The thread blocks can meanwhile run in an asynchronization mode, which implicitly guarantees the independence of inter‐RVE computation as featured by the hierarchical multiscale structure. The parallel computing algorithms are formulated and implemented in an in‐house code, GoDEM, involving the GPU‐specific techniques such as coalesced access, shared memory utilization, and unified memory implementation. Benchmark and performance tests are conducted against an open‐source CPU‐based DEM code under three typical loading conditions. The performance of GoDEM is examined with varying thread‐block size and register pressure of the GPU, and RVE number. It reveals that increasing GPU occupancy by decreasing register pressure results in a significant degradation rather than improvement in performance. We further demonstrate that the proposed GPU parallelism framework may achieve a saturated speedup of approximately 350 compared with the single‐CPU‐core code. As a demonstration on its application for multiscale modeling of granular media, the material point method is coupled with the new framework powered DEM to simulate a typical engineering‐scale problem involving tens of millions of total particles having to be handled. It demonstrates that a speedup of approximately 91 can be achieved by using the proposed framework, compared with the performance of a similar CPU program running on a cluster node of 44 parallel threads. The study offers a viable future solution to large‐scale and multiscale modeling of granular media.

Enabling Latency-Aware Data Initialization for Integrated CPU/GPU Heterogeneous Platform

Nowadays, driven by the needs of autonomous driving and edge intelligence, integrated CPU/GPU heterogeneous platform has gained significant attention from both academia and industry. As the representative series, NVIDIA Jetson family perform well in terms of computation capability, power consumption, and mobile size. Even so, the integrated heterogeneous platform only contains one limited physical memory, which is shared by the CPU and GPU cores and can be the performance bottleneck of the mobile/edge applications. On the other hand, with the unified memory (UM) model introduced in GPU programming, not only the memory allocation is significantly reduced, which mitigates the memory bottleneck of the integrated platforms but also the memory management and programming are simplified. However, as a programming legacy, the UM model still follows the conventional copy-then-execute model, initializing data on the CPU side after allocating memory. This legacy programming mode not only causes significant initialization latency but also slows the execution of the following kernel. In this article, we propose a framework to enable the latency-aware data initialization on the integrated heterogeneous platform. The framework not only includes three data initialization modes, the CPU initialization, GPU initialization, and hybrid initialization, but also utilizes an affinity estimation model to wisely decide the best initialization mode for an application such that the initialization latency performance of the application can be optimized. We evaluate our design on NVIDIA TX2 and AGX platforms. The results demonstrate that the framework can accurately select a data initialization mode for a given application to significantly reduce the initialization latency. We envision this latency-aware data initialization framework being adopted in a full-version of autonomous solution (e.g., Autoware) in the future.