Memory Hierarchy Design Research Articles

Main memory controller with multiple media technologies for big data workloads

SRAM and DRAM memory technologies have been dominant in the implementations of memory subsystems. In recent years, and mainly driven by the huge memory demands of big data applications, NVRAM technology has emerged as a denser memory technology, enabling the design of new hybrid DRAM/NVRAM memory hierarchies that combine multiple memory media technologies to balance memory capacity, latency, cost, and endurance. Two main approaches are being applied to the design of hybrid memory hierarchies: the hybrid address space approach, which relies on the programmer or the operating system to choose the memory technology where each memory page should be stored; and the (only) NVM address space approach, where a faster technology (e.g. commodity DRAM) is needed to acts as a cache of NVRAM to boost the performance. This approach presents architectural challenges such the organization of metadata (e.g. cache tags) and the selection of the proper technology for each memory component. In contrast to existing approaches, this work proposes a memory controller that leverages novel memory technologies such as eDRAM and MRAM to mitigate NVRAM bus contention and improve the performance of the NVM address space. The devised solution proposes a two-level cache hierarchy in the memory controller: a SRAM sector cache and a (x)RAM cache. The (x)RAM cache, much denser, helps significantly reduce the number of accesses to NVRAM. Experimental results show that implementing the (x)RAM cache with eDRAM or MRAM is the best performing approach. Moreover, the eRAM is able to improve the SRAM cache miss penalty by up to 50% and 80%, and overall system performance by 15% and 23%.

GraphAttack

Graph structures are a natural representation of important and pervasive data. While graph applications have significant parallelism, their characteristic pointer indirect loads to neighbor data hinder scalability to large datasets on multicore systems. A scalable and efficient system must tolerate latency while leveraging data parallelism across millions of vertices. Modern Out-of-Order (OoO) cores inherently tolerate a fraction of long latencies, but become clogged when running severely memory-bound applications. Combined with large power/area footprints, this limits their parallel scaling potential and, consequently, the gains that existing software frameworks can achieve. Conversely, accelerator and memory hierarchy designs provide performant hardware specializations, but cannot support diverse application demands. To address these shortcomings, we present GraphAttack, a hardware-software data supply approach that accelerates graph applications on in-order multicore architectures. GraphAttack proposes compiler passes to (1) identify idiomatic long-latency loads and (2) slice programs along these loads into data Producer/ Consumer threads to map onto pairs of parallel cores. Each pair shares a communication queue; the Producer asynchronously issues long-latency loads, whose results are buffered in the queue and used by the Consumer. This scheme drastically increases memory-level parallelism (MLP) to mitigate latency bottlenecks. In equal-area comparisons, GraphAttack outperforms OoO cores, do-all parallelism, prefetching, and prior decoupling approaches, achieving a 2.87× speedup and 8.61× gain in energy efficiency across a range of graph applications. These improvements scale; GraphAttack achieves a 3× speedup over 64 parallel cores. Lastly, it has pragmatic design principles; it enhances in-order architectures that are gaining increasing open-source support.

ZigZag: Enlarging Joint Architecture-Mapping Design Space Exploration for DNN Accelerators

Building efficient embedded deep learning systems requires a tight co-design between DNN algorithms, hardware, and algorithm-to-hardware mapping, a.k.a. dataflow. However, owing to the large joint design space, finding an optimal solution through physical implementation becomes infeasible. To tackle this problem, several design space exploration (DSE) frameworks have emerged recently, yet they either suffer from long runtimes or a limited exploration space. This article introduces ZigZag, a rapid DSE framework for DNN accelerator architecture and mapping. ZigZag extends the common DSE with uneven mapping opportunities and smart mapping search strategies. Uneven mapping decouples operands (W/I/O), memory hierarchy, and mappings (temporal/spatial), opening up a whole new space for DSE, and thus better design points are found by ZigZag compared to other SotAs. For this, ZigZag uses an enhanced nested-for-loop format as a uniform representation to integrate algorithm, accelerator, and algorithm-to-accelerator mapping. ZigZag consists of three key components: 1) an analytical energy-performance-area Hardware Cost Estimator, 2) two Mapping Search Engines that support spatial and temporal even/uneven mapping search, and 3) an Architecture Generator that auto-explores the wide memory hierarchy design space. Benchmarking experiments against published works, in-house accelerator, and existing DSE frameworks, together with three case studies, show the reliability and capability of ZigZag. Up to 64 percent more energy-efficient solutions are found compared to other SotAs, due to ZigZag's uneven mapping capabilities.

Polyhedral Compilation for Racetrack Memories

Traditional memory hierarchy designs, primarily based on SRAM and DRAM, become increasingly unsuitable to meet the performance, energy, bandwidth, and area requirements of modern embedded and high-performance computer systems. Racetrack memory (RTM), an emerging nonvolatile memory technology, promises to meet these conflicting demands by offering simultaneously high speed, higher density, and nonvolatility. RTM provides these efficiency gains by not providing immediate access to all storage locations, but by instead storing data sequentially in the equivalent to nanoscale tapes called tracks . Before any data can be accessed, explicit shift operations must be issued that cost energy and increase access latency. The result is a fundamental change in memory performance behavior: the address distance between subsequent memory accesses now has a linear effect on memory performance. While there are first techniques to optimize programs for linear-latency memories, such as RTM, existing automatic solutions treat only scalar memory accesses. This work presents the first automatic compilation framework that optimizes static loop programs over arrays for linear-latency memories. We extend the polyhedral compilation framework Polly to generate code that maximizes accesses to the same or consecutive locations, thereby minimizing the number of shifts. Our experimental results show that the optimized code incurs up to 85% fewer shifts (average 41%), improving both performance and energy consumption by an average of 17.9% and 39.8%, respectively. Our results show that automatic techniques make it possible to effectively program linear-latency memory architectures such as RTM.

Implementation of DNNs on IoT devices

Driven by the recent growth in the fields of internet of things (IoT) and deep neural networks (DNNs), DNN-powered IoT devices are expected to transform a variety of industrial applications. DNNs, however, involve many parameters and operations to process the data generated by IoT devices. This results in high data-processing latency and energy consumption. New approaches are thus being souhgt to tackle these issues and deploy real-time DNNs into resource-limited IoT devices. This paper presents a comprehensive review on hardware-and-software-co-design approaches developed to implement DNNs on low-resource hardware platforms. These approaches explore the trade-off between energy consumption, speed, classification accuracy, and model size. First, an overview of DNNs is given. Next, available tools for implementing DNNs on low-resource hardware platforms are described. Then, the memory hierarchy designs together with dataflow mapping strategies are presented. Furthermore, various model optimization approaches, including pruning and quantization, are discussed. In addition, case studies are given to demonstrate the feasibility of implementing DNNs for IoT applications. Finally, detailed discussions, research gaps, and future directions are provided. The presented review can guide the design and implementation of the next generation of hardware and software solutions for real-world IoT applications.

Step-by-Step Design of memory hierarchy for heterogeneously-integrated SCM/NAND flash storage

This paper presents the Step-by-Step Design of memory hierarchy for heterogeneously-integrated SCM/NAND flash storage. The heterogeneously-integrated storages utilize multiple types of non-volatile memories such as storage class memories (SCMs) and NAND flash memories. SCMs are further classified to memory-type and storage-type. NAND flash memories are also characterized by the number of stored bits per cell. These non-volatile memories show complex trade-off of various latency, power, capacity and cost. What's worse, in NAND flash, garbage collection seriously degrades the performance. Thus, the intelligent mix & match of these memories is required, depending on each workload. For various storage applications, this paper shows that optimal SCM/NAND flash hierarchies are determined by considering system performance, energy consumption, endurance, and cost for each workload. The recommended non-volatile memory combinations with SCMs and NAND flash memories are proposed for representative 7 storage application workloads with diverse characteristics. The proposed Step-by-Step Design is the useful solution for the heterogeneously-integrated storage with multiple types of non-volatile memories and for various applications.

Тенденции развития вычислительных узлов современных суперкомпьютеров

This work includes analysis of the computation nodes of modern supercomputers from two perspectives; first the hardware components focus and secondly discuss the infrastructure. Identified trends leads to basic options of computation node design. The paper classifies the modern architectures of universal processing cores and specialized hardware accelerators cores; studies the recent trends in memory hierarchy design and intra-node interconnect; the paper includes ways of using the non-volatile memory in modern memory hierarchy. Furthermore, the paper analyses the recent trends in HPC infrastructure, in particular in modern liquid cooling approaches and monitoring. The basic variants of HPC computing nodes design are based on energy efficient universal processor and set of energy-efficient specialized hardware accelerators cores, according to the observed trends. The paper focuses on recent technologies that are currently at various stages of production or at the functional prototype stage. The study also discusses state-of-the-art computational challenges and algorithms-to-architecture mapping issues. Lastly, the paper discusses the current technological problems and main areas to maintain the progress in HPC area.

Reference Capabilities for Safe Parallel Array Programming

The array is a fundamental data structure that provides an efficient way to store and retrieve non-sparse data contiguous in memory. Arrays are important for the performance of many memory-intensive applications due to the design of modern memory hierarchies: contiguous storage facilitates spatial locality and predictive access patterns which enables prefetching. Operations on large arrays often lend themselves well to parallelisation, such as a fork-join style divide-and-conquer algorithm for sorting. For parallel operations on arrays to be deterministic, data-race freedom must be guaranteed. For operations on arrays of primitive data, data-race freedom is obtained by coordinating accesses so that no two threads operate on the same array indices. This is however not enough for arrays of non-primitives due to aliasing: accesses of separate array elements may return pointers to the same object, or overlapping structures. Reference capabilities have been used successfully in the past to statically guarantee the absence of data-races in object-oriented programs. This paper presents the first extension of reference capabilities -- called array capabilities -- that support concurrent and parallel operations on arrays of both primitive and non-primitive values. In addition to element access, array capabilities support the abstract manipulation of arrays, logical splitting of arrays into subarrays, and merging subarrays. These operations allow expressing a wide range of array use cases. This paper presents the array capability design space and show how it applies to a number of array use cases. The core ideas are formalised and proven sound in a simple calculus, along with a proof that shows that well-typed programs with array capabilities are free from data-races.

Memory that never forgets: emerging nonvolatile memory and the implication for architecture design

In order to mitigate the problem of the memory wall, various emerging nonvolatile memory (NVM) technologies have been proposed to replace the traditional ones. These emerging NVMs include spin torque transfer random access memory, PCRAM, resistive random access memory, racetrack memory, etc. Compared to traditional memory technologies, they have the advantages of near-zero standby power, high storage density and nonvolatility, which make them competitive for future memory hierarchy design. However, it is inefficient to directly apply these NVMs in existing memory architectures. On the one hand, these NVMs have their own limitations, such as long write latency, high write energy, limited write numbers, etc. Thus, proper architecture modification is required to adopt them into the traditional memory hierarchy. On the other hand, the unique features of these NVMs allow new memory architectures in memory subsystems and also introduce new challenges to be solved at the same time. In this paper, we not only review various emerging NVMs but also study typical related work on these topics to investigate their implications for memory architecture design.

Impact of spintronic memory on multicore cache hierarchy design

Spintronic memory [spin-transfer torque-magnetic random access memory (STT-MRAM)] is an attractive alternative technology to CMOS since it offers higher density and virtually no leakage current. Spintronic memory continues to require higher write energy, however, presenting a challenge to memory hierarchy design when energy consumption is a concern. This study motivates the use of STT-MRAM for the first-level caches of a multicore processor to reduce energy consumption without significantly degrading the performance. The large STT-MRAM first-level cache implementation saves leakage power. Moreover, the use of small level-0 cache regains the performance drop due to STT-MRAM long write latencies. The combination of both reduces the energy-delay product by 65% on average compared with CMOS baseline. The proposed STT hierarchy also shows good scalability over the CMOS with a few benchmarks which scale significantly better. The PARSEC and Splash2 benchmark suites are analysed running on a modern multicore platform, comparing performance, energy consumption and scalability of the spintronic cache system to a CMOS design.

Vertically Integrated Nanowire-Based Unified Memory.

A vertically integrated nanowire-based device for multifunctional unified memory that combine dynamic random access memory (DRAM) and flash memory in a single transistor is demonstrated for the first time. The device utilizes a gate-all-around (GAA) structure that completely surrounds the nanowire; the structure is built on a bulk silicon wafer. A vertically integrated unified memory (VIUM) device composed of five-story channels was fabricated via the one-route all-dry etching process (ORADEP) with reliable reproducibility, stiction-free stability, and high uniformity. In each DRAM and flash memory operation, the five-story VIUM showed a remarkably enhanced sensing current drivability compared with one-story unified memory (UM) characteristics. In addition to each independent memory mode, the switching endurance of the VIUM was evaluated in the unified mode, which alternatively activates two memory modes, resulting in an even higher sensing memory window than that of the UM. In addition to our previous work on a logic transistor joining high performance with good scalability, this work describes a novel memory hierarchy design with high functionality for system-on-chip (SoC) architectures, demonstrating the practicality and versatility of the vertically integrated nanowire configuration for use in various applications.

Fine grained, direct access file system support for storage class memory

We propose a file system-independent block device interface for SCM, which provides backward compatibility and transparently supports the integration of SCM in current computer systems.We bypass traditional disk caches to allow fast direct I/O access to SCM, so that secondary storage can be accessed identically as main memory.We propose a new hybrid CPU mode for the architecture and introduce two new privileged instructions to provide fine-grained, physical access to SCM. New storage class memory (SCM) technologies, such as phase change memory (PCM) and memristors, are not only byte-addressable like DRAM but also non-volatile like traditional hard disk drives. SCM modules can be placed side-by-side with DRAM on the memory bus, available to memory instructions issued by the CPU. This shift thus engenders a new DRAM-SCM storage architecture, which promises near-DRAM secondary storage access speed at several orders of magnitude faster than magnetic disk or flash memory. Utilizing SCM as a secondary storage device will have a profound impact on memory hierarchy design, requiring new architectural and operating system support.In this paper, we adopt PCM in the storage architecture and present BSS to provide file system-independent Block device Support for Storage class memory. To ensure backward compatibility and high performance, BSS provides a block device interface found in traditional hard disk drives and allows existing file systems to be built on top of itself without any modifications. BSS is designed to directly access the PCM through memory instructions and bypass traditional disk caches that are intended to reduce seek time.The architecture and BSS are prototyped in QEMU and the Linux kernel, respectively. Validation using benchmarks reveals that both work together well to exploit significant advantages of SCM. Compared to traditional hard disk drives, our approach boosts the write/read performance by up to 204x for large files and achieves comparable performance for small ones.

Novel Nonvolatile L1/L2/L3 Cache Memory Hierarchy Using Nonvolatile-SRAM With Voltage-Induced Magnetization Switching and Ultra Low-Write-Energy MTJ

To reduce power consumption of CPU, nonvolatile cache memory has been expected by replacing conventional volatile cache memory based on SRAM. This paper describes nonvolatile cache memory hierarchy design using fast and low-power perpendicular (FL-p-) STT-MRAM. For L3, L2 and L1 cache, 1T-1MTJ with FL-p-STT-MRAM, 6T-2MTJ, and short write pulse based 6T-2MTJ having voltage-induced magnetization switching has been presented for the most suitable combination for the cache memory.

A relation-exchanging buffering mechanism for instruction and data streaming

The traditional memory hierarchy design can smooth the data stream and instruction stream. However, the bandwidth of the instruction stream and data stream are still the main challenge for high-performance microprocessor systems. To improve the data and instruction fetchers, the proposed buffering architecture can exploits both the temporal and spatial localities with a relation-exchanging buffering mechanism. On buffers hit, the dynamic sequences of instruction or data can be reused. At the same time, the prefetching mechanism will be enabled to prefetch the instruction/data being used in the near future. According to the simulation results, the proposed buffering mechanism with the depth 3 and 64-byte line size, which only needs extra 4% hardware cost, is a cost-ineffectiveness choice. The hit rate of an ABP buffer can 22% outperforms that of loop buffer architecture to fetch instruction stream and 7% outperform that of FIFO strategy to fetch data stream.

Moguls

In recent years, the increasing number of processor cores and limited increases in main memory bandwidth have led to the problem of the bandwidth wall , where memory bandwidth is becoming a performance bottleneck. This is especially true for emerging latency-insensitive, bandwidth-sensitive applications. Designing the memory hierarchy for a platform with an emphasis on maximizing bandwidth within a fixed power budget becomes one of the key challenges. To facilitate architects to quickly explore the design space of memory hierarchies, we propose an analytical performance model called Moguls. The Moguls model estimates the performance of an application on a system, using the bandwidth demand of the application for a range of cache capacities and the bandwidth provided by the system with those capacities. We show how to extend this model with appropriate approximations to optimize a cache hierarchy under a power constraint. The results show how many levels of cache should be designed, and what the capacity, bandwidth, and technology of each level should be. In addition, we study memory hierarchy design with hybrid memory technologies, which shows the benefits of using multiple technologies for future computing systems.

A Comprehensive Memory Modeling Tool and Its Application to the Design and Analysis of Future Memory Hierarchies

In this paper we introduce CACTI-D, a significant enhancement of CACTI 5.0. CACTI-D adds support for modeling of commodity DRAM technology and support for main memory DRAM chip organization. CACTI-D enables modeling of the complete memory hierarchy with consistent models all the way from SRAM based L1 caches through main memory DRAMs on DIMMs. We illustrate the potential applicability of CACTI-D in the design and analysis of future memory hierarchies by carrying out a last level cache study for a multicore multithreaded architecture at the 32nm technology node. In this study we use CACTI-D to model all components of the memory hierarchy including L1, L2, last level SRAM, logic process based DRAM or commodity DRAM L3 caches, and main memory DRAM chips. We carry out architectural simulation using benchmarks with large data sets and present results of their execution time, breakdown of power in the memory hierarchy, and system energy-delay product for the different system configurations. We find that commodity DRAM technology is most attractive for stacked last level caches, with significantly lower energy-delay products.

Quantum Memory Hierarchies

The assumption of maximum parallelism support for the successful realization of scalable quantum computers has led to homogeneous, "sea-of-qubits" architectures. The resulting architectures overcome the primary challenges of reliability and scalability at the cost of physically unacceptable system area. We find that by exploiting the natural serialization at both the application and the physical microarchitecture level of a quantum computer, we can reduce the area requirement while improving performance. In particular we present a scalable quantum architecture design that employs specialization of the system into memory and computational regions, each individually optimized to match hardware support to the available parallelism. Through careful application and system analysis, we find that our new architecture can yield up to a factor of thirteen savings in area due to specialization. In addition, by providing a memory hierarchy design for quantum computers, we can increase time performance by a factor of eight. This result brings us closer to the realization of a quantum processor that can solve meaningful problems.

Design and optimization of large size and low overhead off-chip caches

Large off-chip L3 caches can significantly improve the performance of memory-intensive applications. However, conventional L3 SRAM caches are facing two issues as those applications require increasingly large caches. First, an SRAM cache has a limited size due to the low density and high cost of SRAM and, thus, cannot hold the working sets of many memory-intensive applications. Second, since the tag checking overhead of large caches is nontrivial, the existence of L3 caches increases the cache miss penalty and may even harm the performance of some memory-intensive applications. To address these two issues, we present a new memory hierarchy design that uses cached DRAM to construct a large size and low overhead off-chip cache. The high density DRAM portion in the cached DRAM can hold large working sets, while the small SRAM portion exploits the spatial locality appearing in L2 miss streams to reduce the access latency. The L3 tag array is placed off-chip with the data array, minimizing the area overhead on the processor for L3 cache, while a small tag cache is placed on-chip, effectively removing the off-chip tag access overhead. A prediction technique accurately predicts the hit/miss status of an access to the cached DRAM, further reducing the access latency. Conducting execution-driven simulations for a 2 GHz 4-way issue processor and with 11 memory-intensive programs from the SPEC 2000 benchmark, we show that a system with a cached DRAM of 64 MB DRAM and 128 KB on-chip SRAM cache as the off-chip cache outperforms the same system with an 8 MB SRAM L3 off-chip cache by up to 78 percent measured by the total execution time. The average speedup of the system with the cached-DRAM off-chip cache is 25 percent over the system with the L3 SRAM cache.

A separated bit-line unified cache: Conciliating small on-chip cache die-area and low miss ratio

This paper describes an on-chip cache, called a separated bit-line unified cache, which minimizes the chip-area cost in high-performance microprocessors. This unified cache has two ports; one for the instruction bus and the other for the data bus. A separated bit-line memory hierarchy architecture realizes memory hierarchy design with only 10%-20% area overhead. The total cache area can be reduced by more than 20%-30% on the average at capacities of larger than 64 KB with the same hit rate as the conventional cache. The cache latency reaches 4.2 ns at a supply voltage of 1 V. Additionally, the cache is physically addressable even if the cache has a large capacity.

Speculative Memory Cloaking and Bypassing

We revisit memory hierarchy design viewing memory as an inter-operation communication mechanism. We show how dynamically collected information about inter-operation memory communication can be used to improve memory latency. We propose two techniques: (1) Speculative Memory Cloaking, and (2) Speculative Memory Bypassing. In the first technique, we use memory dependence prediction to speculatively identify dependent loads and stores early in the pipeline. These instructions may then communicate prior to address calculation and disambiguation via a fast communication mechanism. In the second technique, we use memory dependence prediction to speculatively transform DEF-store-load-USE dependence chains within the instruction window into DEF-USE ones. As a result, dependent stores and loads are taken off the communication path resulting in further reduction in communication latency. Experimental analysis shows that our methods, on the average, correctly handle 40% (integer) and 19% (floating point) of all memory loads. Moreover, our techniques result in performance improvements of 4.28% (integer) and 3.20% (floating point) over a highly aggressive, dynamically scheduled processor implementing naive memory dependence speculation. We also study the value and address locality characteristics of the values our methods correctly handle. We demonstrate that our methods are orthogonal to both address and value prediction.