Demand Paging Research Articles

Fine-grain Quantitative Analysis of Demand Paging in Unified Virtual Memory

The abstraction of a shared memory space over separate CPU and GPU memory domains has eased the burden of portability for many HPC codebases. However, users pay for ease of use provided by system-managed memory with a moderate-to-high performance overhead. NVIDIA Unified Virtual Memory (UVM) is currently the primary real-world implementation of such abstraction and offers a functionally equivalent testbed for in-depth performance study for both UVM and future Linux Heterogeneous Memory Management (HMM) compatible systems. The continued advocacy for UVM and HMM motivates improvement of the underlying system. We focus on UVM-based systems and investigate the root causes of UVM overhead, a non-trivial task due to complex interactions of multiple hardware and software constituents and the desired cost granularity. In our prior work, we delved deeply into UVM system architecture and showed internal behaviors of page fault servicing in batches. We provided quantitative evaluation of batch handling for various applications under different scenarios, including prefetching and oversubscription. We revealed that the driver workload depends on the interactions among application access patterns, GPU hardware constraints, and host OS components. Host OS components have significant overhead present across implementations, warranting close attention. This extension furthers our prior study in three aspects: fine-grain cost analysis and breakdown, extension to multiple GPUs, and investigation of platforms with different GPU-GPU interconnects. We take a top-down approach to quantitative batch analysis and uncover how constituent component costs accumulate and overlap, governed by synchronous and asynchronous operations. Our multi-GPU analysis shows reduced cost of GPU-GPU batch workloads compared to CPU-GPU workloads. We further demonstrate that while specialized interconnects, NVLink, can improve batch cost, their benefits are limited by host OS software overhead and GPU oversubscription. This study serves as a proxy for future shared memory systems, such as those that interface with HMM, and the development of interconnects.

BARM: A Batch-Aware Resource Manager for Boosting Multiple Neural Networks Inference on GPUs With Memory Oversubscription

Modern intelligent devices usually execute multiple neural networks to improve service quality. However, system performance degrades significantly when the working set exceeds the physical memory capability, a phenomenon called memory oversubscription. To support the execution of multiple independent neural networks with limited physical memory, this article explores resource management in GPUs with unified virtual memory and demand paging. We first analyze the relationship between the simultaneous execution of multiple neural networks from streaming multiprocessors (SM) assignment and page fault overhead from memory thrashing. To boost performance by reducing the page fault penalty, we propose a batch-aware resource management approach, BARM, including (1) batch-aware SM resource allocation to increase the batch size and (2) thrashing-preventing memory allocation to eliminate run-time thrashing. The performance of the proposed method was evaluated using a series of workloads, and response latency is reduced significantly over the state-of-the-art page fault prefetcher and batch-aware TLP management. The proposed framework was also implemented on the real platform and evaluated by a case study, and impressive results were obtained.

Exploring Synchronous Page Fault Handling

The advance of nonvolatile memory in storage technology has presented challenges in redefining the ways in handling the main memory and the storage. This work is motivated by the strong demands in effective handling of page faults over ultralow-latency storage devices. In particular, we propose synchronous and asynchronous prefetching strategies to satisfy process executions with different memory demands in supporting of synchronous page fault handling. An adaptive CPU scheduling strategy is also proposed to cope with the needs of processes in maintaining their working sets in the main memory. Six representative benchmarks and applications were evaluated. It was shown that our strategy can effectively save 12.33% of the total execution time and reduce 13.33% of page faults, compared to the conventional demand paging strategy with nearly no sacrificing of process fairness.

A quantitative evaluation of unified memory in GPUs

The introduction of unified memory and demand paging has simplified programming of graphics processing units (GPUs). It has also enabled oversubscribing the memory for a GPU. However, the overhead of page management makes page faults a performance bottleneck. Sometimes the page eviction policy is unable to mitigate performance slowdown caused by page faults and memory oversubscription. On average, eviction policies such as Random and CAR are not competitive with a traditional least recently used (LRU) policy. Other policies, such as CLOCK-Pro, are designed to overcome challenges with LRU, but they only achieve limited speedup. Even enhancing LRU with page walk hit information does not lead to notable performance improvement. Based on these observations, we propose optimization opportunities to mitigate performance degradation caused by page faults and memory oversubscription. These optimization opportunities include an effective page eviction policy that retains LRU’s advantages while addressing LRU’s inability to deal with thrashing access patterns, page prefetch and pre-eviction, memory-aware throttling, and capacity compression.

HPE: Hierarchical Page Eviction Policy for Unified Memory in GPUs

Recent support for unified memory and demand paging has improved graphics processing unit (GPU) programmability and enabled memory oversubscription. However, this support introduces high overhead when page faults occur. Therefore, when the GPU memory fills to capacity, an important issue is how to select eviction candidates. The widely used policy, LRU, and the advanced replacement policies, RRIP and CLOCK-Pro, suffer from inefficiency when dealing with thrashing access patterns. They also incur significant overhead due to managing metadata at page level. In this article, we propose hierarchical page eviction (HPE), a new replacement policy for GPUs with unified memory. Aided by page walk hit information, HPE manages a page set chain dynamically. It uses statistics to classify applications into three categories and selects an appropriate eviction strategy for each category. It also applies dynamic adjustment to switch the eviction strategy when necessary. The simulation results show that, on average, HPE achieves $1.34\times $ and $1.16\times $ speedup (up to $2.81\times $ ) over LRU when the oversubscription rate is 75% and 50%, respectively. HPE also outperforms RRIP and CLOCK-Pro.

Enlarging I/O Size for Faster Loading of Mobile Applications

As the size of mobile applications grows rapidly, the importance of application loading performance is increasingly emphasized in mobile devices. However, current operating systems rely on demand paging to load the working set of applications into memory, which typically generates small size input/outputs (I/Os) that are not handled well by mobile flash storage devices. We propose an aggressive merging scheme, which consists of an explicit application loading method and a series of optimization techniques: I/O reordering, I/O merging, and I/O padding. The key idea behind our scheme is to enlarge I/O size for application loading to increase the effective storage throughput. Experiments show that our scheme effectively increases the average I/O size by $5.6\times $ , leading to 30% reduction of working set loading time.

Mosaic

Contemporary discrete GPUs support rich memory management features such as virtual memory and demand paging. These features simplify GPU programming by providing a virtual address space abstraction similar to CPUs and eliminating manual memory management, but they introduce high performance overheads during (1) address translation and (2) page faults. A GPU relies on high degrees of thread-level parallelism (TLP) to hide memory latency. Address translation can undermine TLP, as a single miss in the translation lookaside buffer (TLB) invokes an expensive serialized page table walk that often stalls multiple threads. Demand paging can also undermine TLP, as multiple threads often stall while they wait for an expensive data transfer over the system I/O (e.g., PCIe) bus when the GPU demands a page. In modern GPUs, we face a trade-off on how the page size used for memory management affects address translation and demand paging. The address translation overhead is lower when we employ a larger page size (e.g., 2MB large pages, compared with conventional 4KB base pages), which increases TLB coverage and thus reduces TLB misses. Conversely, the demand paging overhead is lower when we employ a smaller page size, which decreases the system I/O bus transfer latency. Support for multiple page sizes can help relax the page size trade-off so that address translation and demand paging optimizations work together synergistically. However, existing page coalescing (i.e., merging base pages into a large page) and splintering (i.e., splitting a large page into base pages) policies require costly base page migrations that undermine the benefits multiple page sizes provide. In this paper, we observe that GPGPU applications present an opportunity to support multiple page sizes without costly data migration, as the applications perform most of their memory allocation en masse (i.e., they allocate a large number of base pages at once).We show that this en masse allocation allows us to create intelligent memory allocation policies which ensure that base pages that are contiguous in virtual memory are allocated to contiguous physical memory pages. As a result, coalescing and splintering operations no longer need to migrate base pages.

Efficient Proximity Search with Query Logs

information retrieval technology there are various techniques for fetching data from resources. And that technique also contains various issues. Information retrieval techniques require advanced manipulating schemes which improves keyword search. There are many techniques have been proposed but results get down when large amount data interrupted. In this paper, have tendency to achieve efficient time and space complexities by integrating proximity information. This system improves the performance by using previous searching results. All the previous system consist basic solutions for extracting results and ranking them. Query logs consists the last searching results and use that results for next search. Fuzzy keyword search truly enhance the system usability. Existing system in databases requires to write complete keyword for searching but by using auto-complete scheme it is easy to type less and find more. In this system proper demand paging algorithm is used for finding previous results.

GPUswap

Over the last few years, GPUs have been finding their way into cloud computing platforms, allowing users to benefit from the performance of GPUs at low cost. However, a large portion of the cloud's cost advantage traditionally stems from oversubscription: Cloud providers rent out more resources to their customers than are actually available, expecting that the customers will not actually use all of the promised resources. For GPU memory, this oversubscription is difficult due to the lack of support for demand paging in current GPUs. Therefore, recent approaches to enabling oversubscription of GPU memory resort to software scheduling of GPU kernels -- which has been shown to induce significant runtime overhead in applications even if sufficient GPU memory is available -- to ensure that data is present on the GPU when referenced. In this paper, we present GPUswap, a novel approach to enabling oversubscription of GPU memory that does not rely on software scheduling of GPU kernels. GPUswap uses the GPU's ability to access system RAM directly to extend the GPU's own memory. To that end, GPUswap transparently relocates data from the GPU to system RAM in response to memory pressure. GPUswap ensures that all data is permanently accessible to the GPU and thus allows applications to submit commands to the GPU directly at any time, without the need for software scheduling. Experiments with our prototype implementation show that GPU applications can still execute even with only 20 MB of GPU memory available. In addition, while software scheduling suffers from permanent overhead even with sufficient GPU memory available, our approach executes GPU applications with native performance.

Demand Paging Techniques for Flash Memory Using Compiler Post-Pass Optimizations

In this article, we propose an application-specific demand paging mechanism for low-end embedded systems that have flash memory as secondary storage. These systems are not equipped with virtual memory. A small memory space called an execution buffer is used to page the code of an application. An application-specific page manager manages the buffer. The page manager is automatically generated by a compiler post-pass optimizer and combined with the application image. The post-pass optimizer analyzes the executable image and transforms function call/return instructions into calls to the page manager. As a result, each function in the code can be loaded into the memory on demand at runtime. To minimize the overhead incurred by the demand paging technique, code clustering algorithms are also presented. We evaluate our techniques with ten embedded applications, and our approach can reduce the code memory size by on average 39.5% with less than 10% performance degradation and on average 14% more energy consumption. Our demand paging technique provides embedded system designers with a trade-off control mechanism between the cost, performance, and energy efficiency in designing embedded systems. Embedded system designers can choose the code memory size depending on their cost, energy, and performance requirements.

Dynamic memory mapping delivers additional flexibility to virtual resource management

The Department of Computer Science and Technology, Peking University, Beijing, China, has shown that a novel Dynamic Memory Mapping (DMM) model brings about additional flexibility to virtual resource management, leading to the feature-adjustable design of a virtual machine monitor (VMM). The study is reported in Issue 53 (June, 2010) of SCIENCE CHINA Information Sciences because of its significant research value. Memory is one of the most frequently accessed components in virtual machine (VM) systems. Because a VM’s memory requirement varies according to the running applications, disregarding the dynamic changes can result in suboptimal use of memory resources, which negatively affects the VM’s performance. However, because the infrastructures of current technologies are usually independent of each other, they exhibit poor extensibility, integrity, and maintainability. To improve the flexibility and extensibility of the VMM, we need to implement a dynamic memory management mechanism in the VMM, while preserving the high efficiency of memory accesses from virtual machines. To resolve these problems, this work proposes a DMM model [1]. The DMM model is a low-level memory management mechanism, which allows dynamic change of the mapping between the pseudo-physical memory as seen from VMs and the machine memory, while the virtual machine is running. On one hand, DMM is independent of, yet compatible with, various virtualization architectures, while on the other, it presents a uniform upward interface for supporting high-level memory management policies. As a result, the DMM layer incorporates high-level policies and low-level implementations by making both of them adjustable. In this work, Prof. Wang, Prof. Luo and their group present the principle of the DMM model, and explain the procedures of various memory management policies under this model, such as demand paging, virtual memory and memory sharing. They also implement the DMM model in KVM, an open source VMM. They first designed a memory pool, a set of machine pages provided by the VMM to a particular virtual machine whose size could be expanded or shrunk while operating. To make the model work in a real system, they manipulated a page-level protection mechanism to propagate memory-mapping updates to the shadow page tables, which is the only way for a VM to access its virtualized memory. They also utilized reverse mapping, a data structure that maps a machine page back to all the shadow page table entries that have mapped it, to facilitate the mapping propagation. The DMM model has several advantages over the current memory management mechanism in VMMs. The first is platform independency—the model is defined abstractly, thus is independent of implementations and computer architectures. The second is flexibility—the DMM model provides a uniform interface for integrating advanced memory management policies. Through the general mechanism, they can work together without conflict. Last but not least, the modular and layered design of the DMM reduces the complexity of a VMM’s code base, and therefore improves the security and dependability of the system. A journal reviewer noted:“This paper addresses the inefficiency in the design of current virtual machine monitors. Their approach is novel and systematic, and incurs only minor overheads. The result is of academic significance and practical value”. Another reviewer said, “It enriches and expands the capacity and capability of virtualization. It offers us new methods to deploy and manage large numbers of virtual machines”. A series of papers about virtual machine system optimization written by Prof. Wang, Prof. Luo and their group have been published in SCIENCE CHINA Information Sciences [2], IEEE Cluster [3], ACM SIGOPS Operating System Review [4]. The authors are affiliated to the Institute of Network Computing and Information Systems (NCIS, http://ncis.pku.edu.cn) at Peking University. This institute, led by Prof. LI XiaoMing, conducts research mainly in the fields of high productivity computing, search engine and Web mining, distributed systems, internet and mobile computing, and database technology.

Non-preemptive demand paging technique for NAND flash-based real-time embedded systems

NAND flash memory is utilized as code storage as well as for file system storage in consumer electronics. The demand paging technique for NAND flash code storage can reduce the required main memory space. However, in realtime systems, demand paging may invoke several problems, one such example is unpredictable timing behavior. Moreover, when NAND flash memory is used for both code storage and file system storage, the resource conflict issue needs to be resolved to allow for simultaneous accesses of demand paging and file system requests. This paper addresses several practical problems of NAND flash-based demand paging in real-time embedded systems and proposes a non-preemptive demand paging technique to resolve the resource conflict within non-preemptive critical sections. Experiments on a real mobile phone platform show that the proposed demand paging requires no significant overhead.

Scratchpad Memory Management Techniques for Code in Embedded Systems without an MMU

We propose a code scratchpad memory (SPM) management technique with demand paging for embedded systems that have no memory management unit. Based on profiling information, a postpass optimizer analyzes and optimizes application binaries in a fully automated process. It classifies the code of the application including libraries into three classes based on a mixed integer linear programming formulation: External code is executed directly from the external memory. Pinned code is loaded into the SPM when the application starts and stays in the SPM. Paged code is loaded into/unloaded from the SPM on demand. We evaluate the proposed technique by running 14 embedded applications both on a cycle-accurate ARM processor simulator and an ARM1136JF-S core. On the simulator, the reference case, a four-way set-associative cache, is compared to a direct-mapped cache and an SPM of comparable die area. On average, we observe an improvement of 12 percent in runtime performance and a 21 percent reduction in energy consumption. On the ARM11 board, the reference case run on the 16-KB four-way set-associative cache is compared to the demand paging solution on the 16-KB SPM, optionally supported by the cache. The measured results show both a runtime performance improvement and a reduction of the energy consumption by 23 percent, on average.

Swap space management technique for portable consumer electronics with NAND flash memory

Flash memory has more capacity and a lower price. It makes flash memory more suitable for portable consumer electronics. Portable consumer electronics such as table PC and smart phones use NAND flash memory as a secondary storage because it has many attractive features such as small size, fast access speeds, and light weight. The portable consumer electronics with NAND flash memory exploit a demand paging to run applications, and also use a swapping to extend a limited main memory space. However, if the portable consumer electronics use NAND flash memory as swap space, it should perform garbage collection, which is a time-consuming operation. Therefore, in order to manage swap space efficiently, this work presents a novel garbage collection policy for the portable consumer electronics with a swap system. The proposed policy has three features important in NAND flash memory based swap systems: (1) long endurance of NAND flash memory, (2) quick garbage collection, and (3) low energy consumption.

DMM: A dynamic memory mapping model for virtual machines

Memory virtualization is an important part in the design of virtual machine monitors (VMM). In this paper, we proposed dynamic memory mapping (DMM)model, a mechanism that allows the VMM to change the mapping between a virtual machine’s physical memory and the underlying hardware resource while the virtual machine is running. By utilizing DMM, the VMM can implement many novel memory management policies, such as Demand Paging, Swapping, Ballooning, Memory Sharing and Copy-On-Write, while preserving compatibility with various architectures. The modular and hierarchical property of the DMM model efficiently incorporates the high-level policies and the low-level implementations, leading to a feature-adjustable VMM design. We presented the principle of the DMM model, and explained the procedures of various memory management policies under this model. Also, we implement the DMM model in KVM, an open source VMM. Our evaluation shows that the DMM model is efficient enough to provide the benefits of dynamic memory resource management with little performance impact.

A new method of fast compression of program code for ota updates in consumer devices

This paper presents a technology for over the air (OTA) updating that allows software updates in consumer devices such as mobile phones and car-navigation systems that are connected to networks, and it adopts NAND flash memories and demand paging technologies. Software updates for these kinds of consumer devices are carried out using the binary difference that minimizes the amount of update data, software construction technologies that minimize the rewriting of flash memories, and fast compression technologies that enables rapid rewriting. This paper focuses on byte pair encoding (BPE), which is one of the most efficient methods of compressing program code in consumer devices that adopts NAND flash memories and demand paging technologies. We describe a fast method of compressing BPE and evaluating it.

Energy and Performance Optimization of Demand Paging With OneNAND Flash

New fusion memory devices consisting of multiple heterogeneous memory components in a single die or package offer efficient ways to optimize embedded systems in terms of energy, performance, and cost. Samsung Electronics recently announced the OneNAND fusion memory, in which a NAND flash array is integrated with dual SRAM buffers to provide a nor-type I/O interface. OneNAND has the low cost and large capacity of a NAND flash but also permits eXecution-in-Place (XIP) like a nor flash. The deployment of such devices requires careful system-level resource management because of their impact on energy consumption and performance, and existing memory optimization techniques, such as the demand paging used with NAND flash, may no longer be appropriate for systems with a fusion memory. We introduce a new online demand paging scheme that fully exploits the XIP capability of OneNAND flash by classifying pages as load preferred (residing in the on-chip SRAM) and XIP preferred (accessed directly from the OneNAND flash and discarded after use). This achieves, on average, a 26% reduction in energy consumption and a 19% increase in performance, compared with conventional NAND flash demand paging.

A Universal Online Caching Algorithm Based on Pattern Matching

We present a universal algorithm for the classical online problem of caching or demand paging. We consider the caching problem when the page request sequence is drawn from an unknown probability distribution and the goal is to devise an efficient algorithm whose performance is close to the optimal online algorithm which has full knowledge of the underlying distribution. Most previous works have devised such algorithms for specific classes of distributions with the assumption that the algorithm has full knowledge of the source. In this paper, we present a universal and simple algorithm based on pattern matching for mixing sources (includes Markov sources). The expected performance of our algorithm is within 4+o(1) times the optimal online algorithm (which has full knowledge of the input model and can use unbounded resources).

Replacement and swapping strategy to improve read performance of portable consumer devices using compressed file systems

Compressed file systems are suitable for portable consumer devices with small capacity of storage systems because the contents of files are stored as a compressed form to save the space of storage. The portable consumer devices also exploit demand paging mechanism to cut down their cost and size. However, if the portable consumer devices use the compressed file systems, there are large computational and I/O overheads. When data are requested, the data should be decompressed before they are accessed by application programs. This is the computational overhead of compressed file systems. And also, to decompress only one byte data, several data should be copied to the main memory from the secondary storage. That is the I/O overhead. To overcome these overheads, this work proposes a new replacement and swapping strategy for the portable consumer devices using the compressed file systems, which keeps the frequently accessed data in the main memory as long as possible and stores the uncompressed data of compressed file systems in the swap space so that serves them directly from the swap space. This strategy increases the page hit ratio, and reduces several copying operations and a decompressing operation of compressed file systems. As a result, it could improve the read performance of portable consumer devices. Trace-driven simulations with real traces show that the proposed replacement and swapping strategy performs better than existing replacement and swapping mechanism in terms of the total read I/O performance, page fault ratio, and page fault latency.

SWL

As mobile phones become increasingly multifunctional, the number and size of applications installed in phones are rapidly increasing. Consequently, mobile phones require more hardware resources such as NOR/NAND flash memory and DRAM, and their production cost is accordingly becoming higher. One candidate solution to reduce production cost is demand paging using MMU. However, demand paging causes unpredictably long page fault latency, and as such mobile phone manufacturers are reluctant to deploy this scheme. In this paper, we present a method that reduces the long latency of page faults by performing page fault handling in a parallelized manner, considering the characteristics of NAND-Type flash memory. We also discuss how to modify the existing page cache replacement policies so that they can exploit the benefits of the parallelized page fault handler. Experimental results show that the parallelized page fault handler improves the worst case latency of page faults significantly, by up to roughly 20%, and that the modified page cache replacement policies improve both the average and worst instruction fetch time.