Dynamic Memory Management Research Articles

DELTA: Memory-Efficient Training via Dynamic Fine-Grained Recomputation and Swapping

To accommodate the increasingly large-scale models within limited-capacity GPU memory, various coarse-grained techniques, such as recomputation and swapping, have been proposed to optimize memory usage. However, these methods have encountered limitations, either in terms of inefficient memory reduction or diminished training performance. In response to this, our paper introduces DELTA, an innovative approach for memory-efficient large-scale model training that combines fine-grained memory optimization and prefetching technology to reduce memory usage while maintaining high training throughput concurrently. Initially, we formulate the problem of memory-throughput joint optimization as an easy-solving 0/1 Knapsack problem. Leveraging this formalization, we use an improving polynomial complexity heuristic algorithm to address the problem effectively. Furthermore, we introduce a novel bidirectional prefetching technology into dynamic memory management, which significantly accelerates the model training when compared to relying solely on recomputation or swapping. Finally, DELTA offers users an automated training execution library, eliminating the need for manual configuration or specialized expertise. Experimental results demonstrate the effectiveness of DELTA in reducing GPU memory consumption. Compared to state-of-the-art methods, DELTA achieves substantial memory savings ranging from 40% to 72%, while maintaining comparable convergence performance for various models, including ResNet-50, ResNet-101, and BERT-Large. Notably, DELTA enables the training of GPT2-Large and GPT2-XL with batch sizes increased by 5.5 × and 6 ×, respectively, showcasing its versatility and practicality in enabling large-scale model training on GPU hardware.

On the Performance Intricacies of Persistent Memory Aware Storage Engines

As key components of DBMSs, various storage engines and index structures have been proposed based on incorrect assumptions before PMem hardware is publicly available. Recent studies reveal that there is a significant performance gap in evaluating index structures on real PMem platforms as compared to DRAM-based emulators. However, a comprehensive evaluation for those PMem-aware database storage engines on real PMem hardware is still missing. Meanwhile, dynamic memory management is more important on PMem systems because PMem is slower than DRAM and unfriendly to random small-writes, and ensuring crash-consistency for the metadata of PMem allocators introduces extra overhead. Therefore, it is essential to understand the performance intricacies of PMem-aware database storage engines from the perspective of PMem allocators. This paper presents a systematic evaluation of three PMem-aware database storage engines using representative workloads and a unified benchmarking framework that is integrated with four PMem allocators. Besides the commonly used metrics, the impact of different hardware configurations (such as NUMA and eADR) on performance is also considered. Through in-depth analysis, we reveal caveats and pitfalls on using or designing PMem-aware storage engines and important insights that can serve as guidelines for future development of PMem allocators and other related components.

Parallel Training of Pre-Trained Models via Chunk-Based Dynamic Memory Management

The pre-trained model (PTM) is revolutionizing Artificial Intelligence (AI) technology. However, the hardware requirement of PTM training is prohibitively high, making it a game for a small proportion of people. Therefore, we proposed PatrickStar system to lower the hardware requirements of PTMs and make them accessible to everyone. PatrickStar uses the CPU-GPU heterogeneous memory space to store the model data. Different from existing works, we organize the model data in memory chunks and dynamically distribute them in the heterogeneous memory. Guided by the runtime memory statistics collected in a warm-up iteration, chunks are orchestrated efficiently in heterogeneous memory and generate lower CPU-GPU data transmission volume and higher bandwidth utilization. Symbiosis with the Zero Redundancy Optimizer, PatrickStar scales to multiple GPUs on multiple nodes. % using data parallelism. The system can train tasks on bigger models and larger batch sizes, which cannot be accomplished by existing works. Experimental results show that PatrickStar extends model scales 2.27 and 2.5 times of DeepSpeed, and consistently exhibits significantly higher execution speed. PatricStar also successfully runs the 175B GPT3 training task on a 32 GPU cluster. Our code is publicly available at https://github.com/Tencent/PatrickStar.

Wfspan: Wait-free Dynamic Memory Management

Dynamic memory allocation plays a vital role in modern application programs. Modern lock-free memory allocators based on hardware atomic primitives usually provide good performance. However, threads may starve in these lock-free implementations, leading to unbounded worst-case execution time that is not allowed in real-time embedded systems. This article presents decentralized dynamic memory management, wfspan, based on non-linearizable wait-free lists. It employs a helping mechanism to ensure no starvation in the lock-free implementation. From the perspective of design tradeoff, wfspan guarantees bounded execution steps in both allocation and deallocation procedure, at the cost of increasing bounded worst-case memory footprint. The results of running benchmarks on an x86/64 and an aarch64 machine illustrate that wfspan achieves competitive performance and memory footprint compared to lock-based and lock-free practical memory allocators while showing superior to other allocators in terms of worst-case execution time.

DDrops: Detecting silent packet drops on programmable data plane

Silent packet drops are common in data center networks, and are a major cause of network performance anomalies (NPAs) that have significant impacts on application performance and network management. However, existing solutions using coarse-grained statistics and flow-level telemetry either fail to provide precise location of packet drops or incur large overhead. This paper presents dDrops, a packet-level telemetry based on programmable data plane to detect and retrieve details of silent packet drops immediately when they happen. dDrops can dynamically adapt to varying ratios of silent packet drops for different ports on a switch to improve performance of silent packet drops detection. Moreover, a dynamic memory management scheme is also designed to efficiently use the limited memory on the data plane of switch. dDrops has been implemented on both P4 hardware programmable switches (based on Intel Tofino ASIC) and BMv2. Extensive experiment results show that dDrops is able to detect and locate the silent packet drops within 5 ms (including detailed information of dropped packets), and reduce the memory consumption by up to 50%.

Dynamic Optimization of On-Chip Memories for HLS Targeting Many-Accelerator Platforms

Many-accelerator platforms have been introduced for maximizing FPGA's throughput. However, as the high saturation rate of the FPGA's on-chip memories limits the number of synthesized accelerators, frameworks for Dynamic Memory Management (DMM) that allow the synthesized designs to allocate/de-allocate on-chip memory resources during run-time have been suggested. Although, those frameworks manage to increase the accelerators’ density by minimizing the utilized memory resources, the parallel execution of many-accelerators may cause severe memory fragmentation and thus memory allocation failures. In this work, a framework that optimizes the memory usage by performing memory defragmentation operations in HLS many-accelerator architectures that share on-chip memories is proposed. Experimental results highlight the effectiveness of the proposed solution to eliminate memory allocation failures due to memory fragmentation, reduce memory allocation failures up to 32% on average and decrease the memory size requirements up to 5% with controllable latency and resource utilization overhead.

Dynamic Memory Management in Massively Parallel Systems: A Case on GPUs.

Due to the high level of parallelism, there are unique challenges in developing system software on massively parallel hardware such as GPUs. One such challenge is designing a dynamic memory allocator whose task is to allocate memory chunks to requesting threads at runtime. State-of-the-art GPU memory allocators maintain a global data structure holding metadata to facilitate allocation/deallocation. However, the centralized data structure can easily become a bottleneck in a massively parallel system. In this paper, we present a novel approach for designing dynamic memory allocation without a centralized data structure. The core idea is to let threads follow a random search procedure to locate free pages. Then we further extend to more advanced designs and algorithms that can achieve an order of magnitude improvement over the basic idea. We present mathematical proofs to demonstrate that (1) the basic random search design achieves asymptotically lower latency than the traditional queue-based design and (2) the advanced designs achieve significant improvement over the basic idea. Extensive experiments show consistency to our mathematical models and demonstrate that our solutions can achieve up to two orders of magnitude improvement in latency over the best-known existing solutions.

An interactive and dynamic scratchpad memory management strategy for multi-core processors

An interactive and dynamic Scratchpad memory (Scratchpad) management approach is proposed in this research manuscript to target multi-core processors. By using MMU an integrated (memory management unit), it can be useful for an existing embedded system that maps into a virtual space the physically addressed Scratchpad memory. A hardware Unit MRSU (Memory Reference Sampling Unit) is introduced depend on mass-count disparity, with a very low probability that models the memory reference data stream. One of the memory addresses is considered as a captured address that is present in a repeatedly referenced memory segment. MRSU generated a hardware interference and the captured / identified repeatedly accessed memory block is placed, through the software, into the Scratchpad space. The page table is also modified by the software so that the Scratchpad memory data block will be forwarded by the follow-up memory accesses. In multi-core processor for Scratchpad management, this proposed strategy is specifically adequate without depending on profiling information and compiler. By profiling or static analysis, the memory accesses behaviour cannot be guessed and a RTOS (Real-Time Operating System) is mostly presented in such an environment. By executing numerous operations on a small Real-Time Operating System the proposed Scratchpad allocation strategy is evaluated with preemptive scheduling. In contrast with a referenced cache-only system with just 1% performance degradation at runtime, in terms of energy consumption on average, this proposed method can achieve an 11% reduction which can be observed in experimental results.

VPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN Training

The increasing computational complexity of DNNs achieved unprecedented successes in various areas such as machine vision and natural language processing (NLP), e.g., the recent advanced Transformer has billions of parameters. However, as large-scale DNNs significantly exceed GPU's physical memory limit, they cannot be trained by conventional methods such as data parallelism. Pipeline parallelism that partitions a large DNN into small subnets and trains them on different GPUs is a plausible solution. Unfortunately, the layer partitioning and memory management in existing pipeline parallel systems are fixed during training, making them easily impeded by out-of-memory errors and the GPU under-utilization. These drawbacks amplify when performing neural architecture search (NAS) such as the evolved Transformer, where different network architectures of Transformer needed to be trained repeatedly. vPipe is the first system that transparently provides dynamic layer partitioning and memory management for pipeline parallelism. vPipe has two unique contributions, including (1) an online algorithm for searching a near-optimal layer partitioning and memory management plan, and (2) a live layer migration protocol for re-balancing the layer distribution across a training pipeline. vPipe improved the training throughput of two notable baselines (Pipedream and GPipe) by 61.4-463.4 percent and 24.8-291.3 percent on various large DNNs and training settings.

A High-Speed NMS Coprocessor for Lightweight Ship Detection Algorithm

With the development of artificial intelligence research, the convolutional neural networks (CNN) triumphantly continue to conquer diverse visual tasks, with lots of technical and practical problems emerging. When the detection networks need to meet the high-speed processing requirement and the low power consumption constraint in edge computing, dedicated circuits and algorithm optimization are necessary. In this brief, we propose a non-maximum suppression (NMS) coprocessor that cooperated with a Vision Processor (VP) to speed up the detection. It contains a parallel process unit and dynamic memory management to accelerate NMS processing. Meanwhile, the software workflow provides a comprehensive method to convert the one-stage detection network to our processor in a post-training manner. The experimental results show that our co-design solution reaches 166 FPS to run a MobileNet-SSD, promising to deploy in edge-computing scenarios. Moreover, the detection accuracy on the Airbus dataset is 88.6%, favorable in practical applications.

The Impact of Cache and Dynamic Memory Management in Static Dataflow Applications

Dataflow is a parallel and generic model of computation that is agnostic of the underlying multi/many-core architecture executing it. State-of-the-art frameworks allow fast development of dataflow applications providing memory, communicating, and computing optimizations by design time exploration. However, the frameworks usually do not consider cache memory behavior when generating code. A generally accepted idea is that bigger and multi-level caches improve the performance of applications. This work evaluates such a hypothesis in a broad experiment campaign adopting different multi-core configurations related to the number of cores and cache parameters (size, sharing, controllers). The results show that bigger is not always better, and the foreseen future of more cores and bigger caches do not guarantee software-free better performance for dataflow applications. Additionally, this work investigates the adoption of two memory management strategies for dataflow applications: Copy-on-Write (CoW) and Non-Temporal Memory transfers (NTM). Experimental results addressing state-of-the-art applications show that NTM and CoW can contribute to reduce the execution time to -5.3% and \(-15.8\%\), respectively. CoW, specifically, shows improvements up to -21.8% in energy consumption with -16.8% of average among 22 different cache configurations.

TENSILE: A Tensor Granularity Dynamic GPU Memory Scheduling Method Toward Multiple Dynamic Workloads System

Recently, deep learning has been an area of intense research. However, as a kind of computing-intensive task, deep learning highly relies on the scale of GPU memory, which is usually prohibitive and scarce. Although some extensive works have been proposed for dynamic GPU memory management, they are hard to apply to systems with multiple dynamic workloads, such as in-database machine learning systems. In this paper, we demonstrated TENSILE, a method of managing GPU memory in tensor granularity to reduce the GPU memory peak, considering the multiple dynamic workloads. TENSILE tackled the cold-starting and across-iteration scheduling problem existing in previous works. We implemented TENSILE on a deep learning framework built by ourselves and evaluated its performance. The experiment results show that TENSILE can save more GPU memory with less extra overhead than prior works in single and multiple dynamic workloads scenarios.

Permchecker: a toolchain for debugging memory managers with typestate

Dynamic memory managers are a crucial component of almost every modern software system. In addition to implementing efficient allocation and reclamation, memory managers provide the essential abstraction of memory as distinct objects, which underpins the properties of memory safety and type safety. Bugs in memory managers, while not common, are extremely hard to diagnose and fix. One reason is that their implementations often involve tricky pointer calculations, raw memory manipulation, and complex memory state invariants. While these properties are often documented, they are not specified in any precise, machine-checkable form. A second reason is that memory manager bugs can break the client application in bizarre ways that do not immediately implicate the memory manager at all. A third reason is that existing tools for debugging memory errors, such as Memcheck, cannot help because they rely on correct allocation and deallocation information to work. In this paper we present Permchecker, a tool designed specifically to detect and diagnose bugs in memory managers. The key idea in Permchecker is to make the expected structure of the heap explicit by associating typestates with each piece of memory. Typestate captures elements of both type (e.g., page, block, or cell) and state (e.g., allocated, free, or forwarded). Memory manager developers annotate their implementation with information about the expected typestates of memory and how heap operations change those typestates. At runtime, our system tracks the typestates and ensures that each memory access is consistent with the expected typestates. This technique detects errors quickly, before they corrupt the application or the memory manager itself, and it often provides accurate information about the reason for the error. The implementation of Permchecker uses a combination of compile-time annotation and instrumentation, and dynamic binary instrumentation (DBI). Because the overhead of DBI is fairly high, Permchecker is suitable for a testing and debugging setting and not for deployment. It works on a wide variety of existing systems, including explicit malloc/free memory managers and garbage collectors, such as those found in JikesRVM and OpenJDK. Since bugs in these systems are not numerous, we developed a testing methodology in which we automatically inject bugs into the code using bug patterns derived from real bugs. This technique allows us to test Permchecker on hundreds or thousands of buggy variants of the code. We find that Permchecker effectively detects and localizes errors in the vast majority of cases; without it, these bugs result in strange, incorrect behaviors usually long after the actual error occurs.

HMvisor: dynamic hybrid memory management for virtual machines

HMvisor: dynamic hybrid memory management for virtual machines

Randomized C/C++ dynamic memory allocator

Dynamic memory management requires special attention in programming. It should be fast and secure at the same time. This paper proposes a new randomized dynamic memory management algorithm designed to meet these requirements. Randomization is a key feature intended to protect applications from “use-after-free” or similar attacks. At the same time, the state in the algorithm consists only of one pointer, so it does not consume extra memory for itself. However, our algorithm is not a universal solution. It does not solve the memory fragmentation problem and it needs further development and testing.

Benchmarking and learning garbage collection delays for resource‐restricted graphical user interfaces

AbstractTablets, smartphones, and wearables have limited resources. Applications on these devices employ a graphical user interface (GUI) for interaction with users. Language runtimes for GUIs employ dynamic memory management using garbage collection (GC). However, GC policies and algorithms are designed for data centers and cloud computing, but they are not necessarily ideal for resource‐constrained embedded devices. In this article, we present GUI GC, a JavaFX GUI benchmark, which we use to compare the performance of the four GC policies of the Eclipse OpenJ9 Java runtime on a resource‐constrained environment. Overall, our experiments suggest that the default policy Gencon registered significantly lower execution times than its counterparts. The region‐based policy, Balanced, did not fully utilize blocking times; thus, using GUI GC, we conducted experiments with explicit GC invocations that measured significant improvements of up to 13.22% when multiple CPUs were available. Furthermore, we created a second version of GUI GC that expands on the number of controllable load‐stressing dimensions; we conducted a large number of randomly configured experiments to quantify the performance effect that each knob has. Finally, we analyzed our dataset to derive suitable knob configurations for desired runtime, GC, and hardware stress levels.

Low-energy motion estimation memory system with dynamic management.

The digital video coding process imposes severe pressure on memory traffic, leading to considerable power consumption related to frequent DRAM accesses. External off-chip memory demand needs to be minimized by clever architecture/algorithm co-design, thus saving energy and extending battery lifetime during video encoding. To exploit temporal redundancies among neighboring frames, the motion estimation (ME) algorithm searches for good matching between the current block and blocks within reference frames stored in external memory. To save energy during ME, this work performs memory accesses distribution analysis of the test zone search (TZS) ME algorithm and, based on this analysis, proposes both a multi-sector scratchpad memory design and dynamic management for the TZS memory access. Our dynamic memory management, called neighbor management, reduces both static consumption—by employing sector-level power gating—and dynamic consumption—by reducing the number of accesses for ME execution. Additionally, our dynamic management was integrated with two previously proposed solutions: a hardware reference frame compressor and the Level C data reuse scheme (using a scratchpad memory). This system achieves a memory energy consumption savings of 99.8% and, when compared to the baseline solution composed of a reference frame compressor and data reuse scheme, the memory energy consumption was reduced by 44.1% at a cost of just 0.35% loss in coding efficiency, on average. When compared with related works, our system presents better memory bandwidth/energy savings and coding efficiency results.

Multilayer Mapping Kit for Autonomous UAV Navigation

Mapping, as the back-end of perception and the front-end of path planning in the modern UAV navigation system, draws our interest. Considering the requirements of UAV navigation and the features of the current embedded computation platforms, we designed and implemented a novel multilayer mapping framework. In this framework, we divided the map into three layers: awareness, local, and global. The awareness map is constructed in cylindrical coordinate, enabling fast raycasting. The local map is a probability-based volumetric map. The global map adopts dynamic memory management, allocating memory for the active mapping area, and recycling the memory from the inactive mapping area. We implemented this mapping framework in three parallel threads: awareness thread, local-global thread, and visualization thread. Finally, we evaluated the mapping kit in both the simulation environment and the real-world scenario with the vision-based sensors. The framework supports different kinds of map outputs for the global or local path planners. The implementation is open-source for the research community.

DySHARQ: Dynamic Software-Defined Hardware-Managed Queues for Tile-Based Architectures

The recent trend towards tile-based manycore architectures has helped to tackle the memory wall by physically distributing memories and processing nodes. However, this introduced a data-to-task locality challenge and inter-tile communication thus often imposes significant software overhead. Thus, we proposed software-defined hardware-managed SHARQ queues that enable efficient inter-tile communication by leveraging user-defined queues with arbitrarily sized elements. To ensure (remote) processing of queued elements, SHARQ introduces an optional handler task, which is scheduled by hardware on demand. Queue management, intra- and inter-tile data transfer, and handler task invocation are entirely handled by hardware. Only rare tasks, like the dynamic queue creation at run-time, are performed in software. DySHARQ, an extension of SHARQ, enables dynamic and concurrent queue memory management and queue length adjustments to be able to adapt to application and resource requirement changes. The DySHARQ hardware is able to monitor the queue memory requirements at run-time and conditionally schedules a software-defined memory management task. It further optimizes the hardware-software interaction for local queue operations. We integrated DySHARQ into the MPI library used by the NAS benchmarks. The evaluation shows a reduction in execution time by up to 43% (compared to software) for the communication intense IS kernel in a 4 $$\times$$ 4 tile design on an FPGA platform with a total of 80 LEON3 cores. The dynamic memory management reduces the memory footprint by 3.75 $$\times$$ in a 2 $$\times$$ 2 design.

Dynamic Resource Allocation and Memory Management Using Machine Learning for Cloud Environments

Dynamic Resource Allocation and Memory Management Using Machine Learning for Cloud Environments