3D-stacked DRAM Research Articles

A 3D-stack DRAM-based PNM architecture design

The article examines methods for integrating 3D-stacked DRAM with AI logic chips, in order to overcome the memory bandwidth challenges faced in the AI inference of transformer models. The findings indicate that this approach can yield a 9x to 3x reduction in power consumption while maintaining similar performance levels, or alternatively, an 8x improvement in performance with comparable power consumption.

Multi-gate BCAT Structure and Select Word-line Driver in DRAM for Reduction of GIDL

In this article, we evaluate gate induced drain leakage that affects the refresh time of buried cell array transistor DRAM cells. We proposed a multi-gate BCAT structure to minimize gate induced drain leakage and modified the select word-line circuit to operate multi-gate buried cell array transistor by adding only one PMOS. In the multi-gate structure, by changing the gate voltage, the work function of the metal gate was adjusted to effectively mitigate the electric field formed in the drain region by approximately four orders. As an adopting multi-gate structure, band to band tunneling is suppressed and gate induced drain leakage current is reduced. We verified that the dual-gate structure had less leakage current than the poly-Si BCAT using the TCAD simulation. The reduction of leakage according to the number of gates is inferred by confirming the reduction in GIDL of the three-gate structure compared with the dual-gate structure. Furthermore, the SPICE simulation confirmed that the proposed select word-line circuit transmits different optimized voltages to multiple gates when it is off than while transferring the same voltage when on. This structure can also be extended for application to other DRAM structures, such as the vertical structure and 3D-stacked DRAMs.

Near-memory Computing on FPGAs with 3D-stacked Memories: Applications, Architectures, and Optimizations

The near-memory computing (NMC) paradigm has transpired as a promising method for overcoming the memory wall challenges of future computing architectures. Modern systems integrating 3D-stacked DRAM memory can be leveraged to prevent unnecessary data movement between the main memory and the CPU. FPGA vendors have started introducing 3D memories to their products in an effort to remain competitive on bandwidth requirements of modern memory-intensive applications. Recent NMC proposals target various types of data processing workloads such as graph processing, MapReduce, sorting, machine learning, and database analytics. In this article, we conduct a literature survey on previous proposals of NMC systems on FPGAs integrated with 3D memories. By leveraging the high bandwidth offered from such memories together with specifically designed hardware, FPGA architectures have become a competitor to GPU solutions in terms of speed and energy efficiency. Various FPGA-based NMC designs have been proposed with software and hardware optimization methods to achieve high performance and energy efficiency. Our review investigates various aspects of NMC designs such as platforms, architectures, workloads, and tools. We identify the key challenges and open issues with future research directions.

NDP-RANK: Prediction and ranking of NDP systems performance using machine learning

The near-data processing (NDP) paradigm has recently gained popularity as a promising method for mitigating the memory wall challenges of future computing systems. Modern 3D-stacked DRAM memories can be leveraged to prevent unnecessary data movement between the main memory and the CPU. However, identifying which type of NDP system, in terms of processing elements and the type of high performance 3D memory used, is suitable for a given application is not a trivial task. Understanding the interactions between modern workloads and the memory subsystem requires a lot of effort. Each memory type presents its unique advantages and shortcomings. In addition, memory access patterns vary greatly across applications. Therefore, the performance of a given application on a given memory type is difficult to intuitively predict. There is no specific NDP system or memory type that can provide high performance for every application.We propose NDP-RANK, a machine learning framework that can efficiently decide which NDP system is suitable for an application. NDP-RANK’s performance prediction is based on an input set of application and microarchitecture characteristics. We build machine learning models that can be used to conduct design-space exploration of NDP systems and accurately predict performance of previously unseen applications on several NDP architectures. Our models are orders of magnitude faster than architectural simulation. They can accurately predict performance with coefficients of determination ranging between 0.89 and 0.96, and root mean square errors ranging between 0.04 and 0.16.

Performance and Area Trade-Off of 3D-Stacked DRAM Based Chip Multiprocessor with Hybrid Interconnect

Nowadays, the number of cores on a chip multiprocessor is growing to increase system performance. However, inadequate on-chip interconnection and memory bandwidth have diminished the potential of these chip multiprocessors. High performance interconnects, 3D-stacked main memory, and large on-chip caches are the architectural parameters used to tackle this issue. For a fixed die size, high performance interconnects, and the 3D-stacked memory fosters the increasing rate of the number of cores on a chip multiprocessor whereas increasing the size of on-chip cache poses a restriction. In this work, we study the trade-off between the performance and overall chip area (evaluated using the number of cores, their types and cache size per core) of the chip multiprocessor for different combinations of the interconnection network, DRAM memory (off-chip or on-chip) and self-adaptive page mapping. Our experiments show that for the base-case (chip multiprocessor with the off-chip DRAM and without hybrid interconnect) architecture and without page mapping technique, to increase the core count for a fixed die size, merely shrinking the cache size degrades the performance. Whereas, reducing the cache size and increasing the chip multiprocessor core count along with our considered target architecture and page mapping technique scale up the performance.

Memory-Side Prefetching Scheme Incorporating Dynamic Page Mode in 3D-Stacked DRAM

Modern multiprocessor systems running multiple applications concurrently exhibit irregular memory access pattern during different phases of execution. The principle of locality is hard to exploit in the presence of such irregular memory requests and may result in additional delays due to resource conflicts throughout memory hierarchy. Prefetching is a promising technique to reduce the memory access latency where data is speculatively fetched ahead of time and stored in a faster memory structure like cache or dedicated prefetch buffer. The emergence of 3D-stacked DRAM provides huge internal bandwidth that makes memory-side prefetching an effective approach to improving system performance. Leveraging the unique architecture of 3D-stacked DRAM, we introduce a memory-side prefetching scheme that works in conjunction with dynamic page mode to reduce memory access latency. We introduce a novel prefetch buffer management scheme that makes intelligent replacement decision based on the utilization and recency of the prefetched data, which also serves as a guidance for future prefetching. Simulation results indicate that our approach improves performance by 21.8 percent on average, compared to a baseline scheme that prefetches a whole row on consecutive hits and implements static open page policy. Our scheme also outperforms an existing memory-side prefetching scheme by 13.2 percent on average, which dynamically adjusts the prefetch degree based on the usefulness of prefetched data.

A 96-MB 3D-Stacked SRAM Using Inductive Coupling With 0.4-V Transmitter, Termination Scheme and 12:1 SerDes in 40-nm CMOS

A 28.8-GB/s 96-MB 3D-stacked SRAM is presented. A total of eight SRAM dies, designed in a 40-nm CMOS process, are vertically stacked and connected using an inductive coupling wireless link with a low-voltage NMOS push-pull transmitter that reduces the power of the link by 35% with a 0.4-V power supply. The SRAM utilizes an inverted bit insertion scheme that compensates for the degradation of the first transmitted bit, a coil termination scheme that aims to eliminate the ringing of 3D inductive coupling bus, and a 12:1 SerDes that minimizes power consumption and area overhead in inductive coupling channels. Low-power, large-capacity, 3-cycle latency 3D-stacked SRAM for a DNN accelerator is achieved with the combination of these techniques to serve as a replacement of 3D-stacked DRAM. The performance of the proposed 3D-SRAM is compared with HBM DRAM and achieves more than 50% lower energy consumption. The scaling scenario of the SRAM module is discussed in light of the scaling of the inductive coupling technology and logic process.

MemSZ

This article describes Memory Squeeze (MemSZ), a new approach for lossy general-purpose memory compression. MemSZ introduces a low latency, parallel design of the Squeeze (SZ) algorithm offering aggressive compression ratios, up to 16:1 in our implementation. Our compressor is placed between the memory controller and the cache hierarchy of a processor to reduce the memory traffic of applications that tolerate approximations in parts of their data. Thereby, the available off-chip bandwidth is utilized more efficiently improving system performance and energy efficiency. Two alternative multi-core variants of the MemSZ system are described. The first variant has a shared last-level cache (LLC) on the processor-die, which is modified to store both compressed and uncompressed data. The second has a 3D-stacked DRAM cache with larger cache lines that match the granularity of the compressed memory blocks and stores only uncompressed data. For applications that tolerate aggressive approximation in large fractions of their data, MemSZ reduces baseline memory traffic by up to 81%, execution time by up to 62%, and energy costs by up to 25% introducing up to 1.8% error to the application output. Compared to the current state-of-the-art lossy memory compression design, MemSZ improves the execution time, energy, and memory traffic by up to 15%, 9%, and 64%, respectively.

Run-time adaptive data page mapping: A Comparison with 3D-stacked DRAM cache

In the current chip-multiprocessor era, 3D-stacked DRAM became an attractive alternative to mitigate the DRAM bandwidth wall problem. In a chip-multiprocessor, the 3D-stacked DRAM is architect either (a) to cache both local and remote data or (b) to cache only the local data. Caching only local data into the 3D-stacked DRAM enforces the chip-multiprocessors to suffer inter-node latency overhead while accessing remote data. However, caching both local and remote data onto the 3D-stacked DRAM requires a large coherence directory (tens of MBs) to ensure correctness. In this paper, we consider a 3D-stacked DRAM based chip-multiprocessor and perform a comparative study between (a) high level adaptive run-time data page mapping onto DRAM with an auxiliary small SRAM buffer as a performance booster, and (b) DRAM used as coherent cache. Our experiment on a 64 core chip-multiprocessor system with 4GB of 3D-stacked DRAM shows that our adaptive run-time data page mapping on DRAM along with an SRAM buffer outperforms the base-case (where DRAM caches only local data) by an average of 48%. Moreover, our method shows a performance improvement by an average of 40% when compared with a recent state-of-art work (where DRAM caches both local and remote data).

NoM: Network-on-Memory for Inter-Bank Data Transfer in Highly-Banked Memories

Data copy is a widely-used memory operation in many programs and operating system services. In conventional computers, data copy is often carried out by two separate read and write transactions that pass data back and forth between the DRAM chip and the processor chip. Some prior mechanisms propose to avoid this unnecessary data movement by using the shared internal bus in the DRAM chip to directly copy data within the DRAM chip (e.g., between two DRAM banks). While these methods exhibit superior performance compared to conventional techniques, data copy across different DRAM banks is still greatly slower than data copy within the same DRAM bank. Hence, these techniques have limited benefit for the emerging 3D-stacked memories (e.g., HMC and HBM) that contain hundreds of DRAM banks across multiple memory controllers. In this paper, we present Network-on-Memory (NoM), a lightweight inter-bank data communication scheme that enables direct data copy across both memory banks of a 3D-stacked memory. NoM adopts a TDM-based circuit-switching design, where circuit setup is done by the memory controller. Compared to state-of-the-art approaches, NoM enables both fast data copy between multiple DRAM banks and concurrent data transfer operations. Our evaluation shows that NoM improves the performance of data-intensive workloads by 3.8X and 75%, on average, compared to the baseline conventional 3D-stacked DRAM architecture and state-of-the-art techniques, respectively.

Stress-Induced Performance Shifts in 3D DRAMs

3D-stacked DRAMs can significantly increase cell density and bandwidth while also lowering power consumption. However, 3D structures experience significant thermomechanical stress due to the differential rate of contraction of the constituent materials, which have different coefficients of thermal expansion. This impacts circuit performance. This article develops a procedure that performs a performance analysis of 3D DRAMs, capturing the impact of both layout-aware stress and layout-independent stress on parameters such as latency, leakage power, refresh power, area, and bus delay. The approach first proposes a semianalytical stress analysis method for the entire 3D DRAM structure, capturing the stress induced by through-silicon-vias (TSVs), micro bumps, package bumps, and warpage. Next, this stress is translated to variations in device mobility and threshold voltage, after which analytical models for latency, leakage power, and refresh power are derived. Finally, a complete analysis of performance variations is performed for various 3D DRAM layout configurations to assess the impact of layout-dependent stress. We explore the use of alternative flexible package substrate options to mitigate the performance impact of stress. Specifically, we explore the use of an alternative bendable package substrate made of polyimide to reduce warpage-induced stress, and show that it reduces stress-induced variations and improves the performance metrics for stacked 3D DRAMs.

Design of Processing-“Inside”-Memory Optimized for DRAM Behaviors

The computing domain of today’s computer systems is moving very fast from arithmetic to data processing as data volumes grow exponentially. As a result, processing-in-memory (PIM) studies have been actively conducted to support the data processing in or near memory devices to address the limited bandwidth and high power consumption due to data movement between CPU/GPU and memory. However, most PIM studies so far have been conducted in a way that the processing units are designed only as an accelerator on the base die of 3D-stacked DRAM, not involved inside memory while not servicing the standard DRAM requests during the PIM execution. Therefore, in this paper, we show how to design and operate the PIM computing units inside DRAM by effectively coordinating with standard DRAM operations while achieving the full computing performance and minimizing the implementation cost. To make our goals, we extend a standard DRAM state diagram to depict the PIM behaviors in the same way as standard DRAM commands are scheduled and operated on the DRAM devices and exploit several levels of parallelism to overlap memory and computing operations. Also, we present how the entire architecture layers from applications to operating systems, memory controllers, and PIM devices should work together for the effective execution by applying our approaches to our experiment platform. In our HBM2-based experimental platform to include 16-cycle MAC (Multiply-and-Add) units and 8-cycle reducers for a matrix-vector multiplication, we achieved 406% and 35.2% faster performance by the all-bank and the per-bank schedulings, respectively, at ( $1024\times1024$ ) $\times $ ( $1024\times1$ ) 8-bit integer matrix-vector multiplication than the execution of only its operand burst reads assuming the external full DRAM bandwidth. It should be noted that the performance of the PIM on a base die of a 3D-stacked memory cannot be better than that provided by the full bandwidth in any case.

EXTREME: Exploiting Page Table for Reducing Refresh Power of 3D-Stacked DRAM Memory

For future exascale computing systems, ultra-high-density memories would be required that consume low power to process massive data. Of the various memory devices, 3D-stacked DRAMs using TSVs are a perfect solution for this purposes. In addition to providing high capacity, these provide functional flexibility to the computing system by attaching a logic die in each 3D-stacked DRAM chip. However, the high capacity 3D-stacked DRAMs suffer from a significant loss of refresh power, which is solely required to maintain data. Although various schemes have been proposed to mitigate this issue, they cannot be adopted by commercial products due to compatibility issues. To tackle this issue, we propose EXTREME, which effectively reduces the refresh power of 3D-Stacked DRAMs. In order to retain the compatibility with OS, a simple page table manager is implemented at the logic die of 3D-stacked DRAM devices, which pins the page table to a specific memory space. The experiment results demonstrate that this reduces the refresh power at idle time by up to 98 percent with 16 KB of SRAM (static RAM) and 64 KB of DRAM register overhead for a 2 GB 3D-stacked DRAM device.

Toward multi-programmed workloads with different memory footprints: a self-adaptive last level cache scheduling scheme

With the emerging of 3D-stacking technology, the dynamic random-access memoryDRAM) can be stacked on chips to architect the DRAM last level cacheLLC). Compared with static random-access memorySRAM), DRAM is larger but slower. In the existing research papers, a lot of work has been devoted to improving the workload performance using SRAM and stacked DRAM together, ranging from SRAM structure improvement, to optimizing cache tag and data access. Instead, little attention has been paid to designing an LLC scheduling scheme for multi-programmed workloads with different memory footprints. Motivated by this, we propose a self-adaptive LLC scheduling scheme, which allows us to utilize SRAM and 3D-stacked DRAM efficiently, achieving better workload performance. This scheduling scheme employs (1) an evaluation unit, which is used to probe and evaluate the cache information during the process of programs being executed; and linebreak (2) an implementation unit, which is used to self-adaptively choose SRAM or DRAM. To make the scheduling scheme work correctly, we develop a data migration policy. We conduct extensive experiments to evaluate the performance of our proposed scheme. Experimental results show that our method can improve the multi-programmed workload performance by up to 30% compared with the state-of-the-art methods.

CasHMC: A Cycle-Accurate Simulator for Hybrid Memory Cube

3D-stacked DRAM has been actively studied to overcome the limits of conventional DRAM. The Hybrid Memory Cube (HMC) is a type of 3D-stacked DRAM that has drawn great attention because of its usability for server systems and processing-in-memory (PIM) architecture. Since HMC is not directly stacked on the processor die where the central processing units (CPUs) and graphic processing units (GPUs) are integrated, HMC has to be linked to other processor components through high speed serial links. Therefore, the communication bandwidth and latency should be carefully estimated to evaluate the performance of HMC. However, most existing HMC simulators employ only simple HMC modeling. In this paper, we propose a cycle-accurate simulator for hybrid memory cube called CasHMC. It provides a cycle-by-cycle simulation of every module in an HMC and generates analysis results including a bandwidth graph and statistical data. Furthermore, CasHMC is implemented in C++ as a single wrapped object that includes an HMC controller, communication links, and HMC memory. Instantiating this single wrapped object facilitates simultaneous simulation in parallel with other simulators that generate memory access patterns such as a processor simulator or a memory trace generator.

Opportunities of synergistically adjusting voltage-frequency levels of cores and DRAMs in CMPs with 3d-stacked DRAMs for efficient thermal control

Chip-Multiprocessors (CMPs) with 3D-stacked DRAMs is promising for solving the memory wall problem, but the high power density makes 3D ICs frequently operate at or near the thermal limit. System hot spot of a CMP with 3Dstacked DRAMs is usually in DRAMs that are in the layers farthest from the heat sink. Heat from DRAMs and cores are all accumulated in DRAMs. Therefore, existing thermal managements for 3D ICs all perform thermal control on cores only because lowering the power-level of cores can also lower DRAM access frequency. However, as the power consumption of single DRAM access increases with the number of DRAM stacks and the width of the vertical links, the instantaneous DRAM accesses may easily overheat the system. So, in addition to lowering the access frequency of DRAMs, reducing the power consumption per DRAM access is also crucial. In this paper, we characterize the thermal and performance behavior of the target architecture when the voltage and frequency levels of cores and DRAMs are synergistically controlled. We also evaluate the thermal and performance behavior of existing thermal control methods that can be applied to the target architecture. The insights provided by the characterizations presented in this paper are important for developing an effective thermal management policy for CMPs with 3D-stacked DRAMs. Our results show that, synergistically controlling the voltage-frequency levels of cores and DRAMs does achieve higher thermal efficiency than controlling cores only.

Simultaneous Multi-Layer Access

3D-stacked DRAM alleviates the limited memory bandwidth bottleneck that exists in modern systems by leveraging through silicon vias (TSVs) to deliver higher external memory channel bandwidth. Today’s systems, however, cannot fully utilize the higher bandwidth offered by TSVs, due to the limited internal bandwidth within each layer of the 3D-stacked DRAM. We identify that the bottleneck to enabling higher bandwidth in 3D-stacked DRAM is now the global bitline interface , the connection between the DRAM row buffer and the peripheral IO circuits. The global bitline interface consists of a limited and expensive set of wires and structures, called global bitlines and global sense amplifiers , whose high cost makes it difficult to simply scale up the bandwidth of the interface within a single DRAM layer in the 3D stack. We alleviate this bandwidth bottleneck by exploiting the observation that several global bitline interfaces already exist across the multiple DRAM layers in current 3D-stacked designs, but only a fraction of them are enabled at the same time. We propose a new 3D-stacked DRAM architecture, called Simultaneous Multi-Layer Access (SMLA), which increases the internal DRAM bandwidth by accessing multiple DRAM layers concurrently, thus making much greater use of the bandwidth that the TSVs offer. To avoid channel contention, the DRAM layers must coordinate with each other when simultaneously transferring data. We propose two approaches to coordination, both of which deliver four times the bandwidth for a four-layer DRAM, over a baseline that accesses only one layer at a time. Our first approach, Dedicated-IO, statically partitions the TSVs by assigning each layer to a dedicated set of TSVs that operate at a higher frequency. Unfortunately, Dedicated-IO requires a nonuniform design for each layer (increasing manufacturing costs), and its DRAM energy consumption scales linearly with the number of layers. Our second approach, Cascaded-IO, solves both issues by instead time multiplexing all of the TSVs across layers. Cascaded-IO reduces DRAM energy consumption by lowering the operating frequency of higher layers. Our evaluations show that SMLA provides significant performance improvement and energy reduction across a variety of workloads (55%/18% on average for multiprogrammed workloads, respectively) over a baseline 3D-stacked DRAM, with low overhead.

Near-DRAM Acceleration with Single-ISA Heterogeneous Processing in Standard Memory Modules

Energy consumed for transferring data across the processor memory hierarchy constitutes a large fraction of total system energy consumption, and this fraction has steadily increased with technology scaling. This article presents a near-DRAM acceleration (NDA) architecture wherein lightweight processors (LWPs) with the same ISA as their host processor are 3D-stacked atop commodity DRAM devices in a standard memory module to efficiently process data. In contrast to previous architectures, the authors' NDA architecture requires negligible changes to commodity DRAM device and standard memory module architectures. This allows the NDA to be more easily adopted in both existing and emerging systems. Experiments demonstrate that, on average, the authors' NDA-based system consumes almost 65 percent less energy at nearly two times higher performance than the baseline system.

10x10

Customized architecture is widely recognized as an important approach for improved performance and energyefficiency. To balance generality and customization benefit, researchers have proposed to federate heterogeneous micro-engines. Using the 10x10 architecture and an integrated image and vision benchmark as a case study, we explore the performance and energy benefits achievable. Results for current 32nm technology and DDR3 memory show 10x10 architecture benefits of 140x performance and 72x energy overall. Adding 3D-stacked DRAM increase benefits to 171x (performance) and 100x (energy). Finally, considering future 7nm transistor process, benefits as large as 597x (performance) and 137x energy are observed.

FFTs with Near-Optimal Memory Access Through Block Data Layouts: Algorithm, Architecture and Design Automation

Fast Fourier transform algorithms on large data sets achieve poor performance on various platforms because of the inefficient strided memory access patterns. These inefficient access patterns need to be reshaped to achieve high performance implementations. In this paper we formally restructure 1D, 2D and 3D FFTs targeting a generic machine model with a two-level memory hierarchy requiring block data transfers, and derive memory access pattern efficient algorithms using custom block data layouts. These algorithms need to be carefully mapped to the targeted platform's architecture, particularly the memory subsystem, to fully utilize performance and energy efficiency potentials. Using the Kronecker product formalism, we integrate our optimizations into Spiral framework and evaluate a family of DRAM-optimized FFT algorithms and their hardware implementation design space via automated techniques. In our evaluations, we demonstrate DRAM-optimized accelerator designs over a large tradeoff space given various problem (single/double precision 1D, 2D and 3D FFTs) and hardware platform (off-chip DRAM, 3D-stacked DRAM, ASIC, FPGA, etc.) parameters. We show that Spiral generated pareto optimal designs can achieve close to theoretical peak performance of the targeted platform offering 6x and 6.5x system performance and power efficiency improvements respectively over conventional row-column FFT algorithms.