Memory Bottleneck Research Articles

Leveraging the Hardware Resources to Accelerate cryo-EM Reconstruction of RELION on the New Sunway Supercomputer

The fast development of biomolecular structure determination has enabled the fine-grained study of objects in the micro-world, such as proteins and RNAs. The world is benefited. However, as the computational algorithms are constantly developed, the enrichment of features increases the algorithmic complexity and brings more computationally unfriendly modules. It calls for efficient solutions to leverage the rich and various hardware resources from the world’s most state-of-the-art supercomputing systems, and to fully accelerate the performance of the applications. In this paper, we present our efforts on porting and optimizing the 3D reconstruction of RELION, one of the most popular cryo-EM software for biomolecular structure determinations, by leveraging different resources of the latest generation of Sunway heterogeneous supercomputer. Several novel approaches are proposed to resolve different challenges faced by the complex algorithm, including a multi-level parallel scheme and operator optimizations to smartly map and scale RELION, efficient strategies to largely address the memory bottlenecks and improve data locality, lock-free writing solutions to minimize write-write conflicts, and pipelining approaches to obtain excellent computation and communication overlap. Combining all proposed optimizations, the computation time is greatly reduced to under 2 hours, achieving 11.9 × and 8.9 × speedups on two different datasets. The overall design scales to 131,072 cores, increasing parallel efficiency from 33% to 61% and from 46% to 70%, respectively. To the best of our knowledge, this is the first work that fully optimized and scaled the 3D reconstruction of RELION using the latest Sunway system.

A qubit-efficient variational selected configuration-interaction method

Abstract Finding the ground-state energy of molecules is an important and challenging computational problem for which quantum computing can potentially find efficient solutions. The variational quantum eigensolver (VQE) is a quantum algorithm that tackles the molecular groundstate problem and is regarded as one of the flagships of quantum computing. Yet, to date, only very small molecules were computed via VQE, due to high noise levels in current quantum devices. Here we present an alternative variational quantum scheme that requires significantly less qubits than VQE. The reduction in the qubit number allows for shallower circuits to be sufficient, rendering the method more resistant to noise. The proposed algorithm, termed variational quantum selected-configuration-interaction (VQ-SCI), is based on: (a) representing the target groundstate as a superposition of Slater determinant configurations, encoded directly upon the quantum computational basis states; and (b) selecting a-priory only the most dominant configurations. This is demonstrated through a set of groundstate calculations of the H2, LiH, BeH2, H2O, NH3 and C2H4 molecules in the sto-3g basis set, performed on IBM quantum devices. We show that the VQ-SCI reaches the full configuration interaction energy within chemical accuracy using the lowest number of qubits reported to date. Moreover, when the SCI matrix is generated ‘on the fly’, the VQ-SCI requires exponentially less memory than classical SCI methods. This offers a potential remedy to a severe memory bottleneck problem in classical SCI calculations. Finally, the proposed scheme is general and can be straightforwardly applied for finding the groundstate of any Hermitian matrix, outside the chemical context.

Efficient memristor accelerator for transformer self-attention functionality

The adoption of transformer networks has experienced a notable surge in various AI applications. However, the increased computational complexity, stemming primarily from the self-attention mechanism, parallels the manner in which convolution operations constrain the capabilities and speed of convolutional neural networks (CNNs). The self-attention algorithm, specifically the matrix-matrix multiplication (MatMul) operations, demands a substantial amount of memory and computational complexity, thereby restricting the overall performance of the transformer. This paper introduces an efficient hardware accelerator for the transformer network, leveraging memristor-based in-memory computing. The design targets the memory bottleneck associated with MatMul operations in the self-attention process, utilizing approximate analog computation and the highly parallel computations facilitated by the memristor crossbar architecture. Remarkably, this approach resulted in a reduction of approximately 10 times in the number of multiply-accumulate (MAC) operations in transformer networks, while maintaining 95.47% accuracy for the MNIST dataset, as validated by a comprehensive circuit simulator employing NeuroSim 3.0. Simulation outcomes indicate an area utilization of 6895.7 μm2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mu m^2$$\\end{document}, a latency of 15.52 seconds, an energy consumption of 3 mJ, and a leakage power of 59.55 μW\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\mu W$$\\end{document}. The methodology outlined in this paper represents a substantial stride towards a hardware-friendly transformer architecture for edge devices, poised to achieve real-time performance.

Key-Value Cache Quantization in Large Language Models: A Safety Benchmark

The misuse of large language models (LLMs) is an ongoing concern with the expansion of general public access to LLMs. One reason for expanded access is the development of key-value (KV) cache quantization, a technique that significantly reduces both the large computing resource requirements and memory bottlenecks that are characteristic of LLMs. As more developers and vendors begin to prioritize efficiency in LLM training, protective measures against the misuse of language models become more of an afterthought. To address the expected increase of LLM misuse accompanying KV cache quantization, this paper covers a proof-of-concept benchmark to evaluate the proficiency of LLMs in response safety when tested against a sample of unsafe questions consisting of 13 different question categories. Response safety is a model’s ability to both clearly deny providing a response to a given question and avoid providing any additional information that attempts to provide an accurate answer. By testing the sample against the Meta Llama-2-7B pretrained chat model, we determine response safety fine-tuning considerations that address performance bias among the 13 question categories. We hope this study brings attention to not only the sacrifices of accuracy but also response safety of KV cache quantization in large language models. (Code and data are available at https://github.com/TimochiL/llm_benchmark.) Disclaimer: This paper contains examples of harmful language. Reader discretion is recommended.

Mixed-precision computing in the GRIST dynamical core for weather and climate modelling

Abstract. Atmosphere modelling applications are becoming increasingly memory-bound due to the inconsistent development rates between processor speeds and memory bandwidth. In this study, we mitigate memory bottlenecks and reduce the computational load of the Global–Regional Integrated Forecast System (GRIST) dynamical core by adopting a mixed-precision computing strategy. Guided by an application of the iterative development principle, we identify the coded equation terms that are precision insensitive and modify them from double to single precision. The results show that most precision-sensitive terms are predominantly linked to pressure gradient and gravity terms, while most precision-insensitive terms are advective terms. Without using more computing resources, computational time can be saved, and the physical performance of the model is largely kept. In the standard computational test, the reference runtime of the model's dry hydrostatic core, dry nonhydrostatic core, and the tracer transport module is reduced by 24 %, 27 %, and 44 %, respectively. A series of idealized tests, real-world weather and climate modelling tests, was performed to assess the optimized model performance qualitatively and quantitatively. In particular, in the long-term coarse-resolution climate simulation, the precision-induced sensitivity can manifest at the large scale, while in the kilometre-scale weather forecast simulation, the model's sensitivity to the precision level is mainly limited to small-scale features, and the wall-clock time is reduced by 25.5 % from the double- to mixed-precision full-model simulations.

Bi-Directional and Operand-Controllable In-Memory Computing for Boolean Logic and Search Operations with Row and Column Directional SRAM (RC-SRAM).

The von Neumann architecture is no longer sufficient for handling large-scale data. In-memory computing has emerged as the potent method for breaking through the memory bottleneck. A new 10T SRAM bitcell with row and column control lines called RC-SRAM is proposed in this article. The architecture based on RC-SRAM can achieve bi-directional and operand-controllable logic-in-memory and search operations through different signal configurations, which can comprehensively respond to various occasions and needs. Moreover, we propose threshold-controlled logic gates for sensing, which effectively reduces the circuit area and improves accuracy. We validate the RC-SRAM with a 28 nm CMOS technology, and the results show that the circuits are not only full featured and flexible for customization but also have a significant increase in the working frequency. At VDD = 0.9 V and T = 25 °C, the bi-directional search frequency is up to 775 MHz and 567 MHz, and the speeds for row and column Boolean logic reach 759 MHz and 683 MHz.

Fast Loosely-Timed Deep Neural Network Models with Accurate Memory Contention

The emergence of data-intensive applications, such as Deep Neural Networks (DNN), exacerbates the well-known memory bottleneck in computer systems and demands early attention in the design flow. Electronic System-Level (ESL) design using SystemC Transaction Level Modeling (TLM) enables effective performance estimation, design space exploration (DSE), and gradual refinement. However, memory contention is often only detectable after detailed TLM-2.0 approximately-timed or cycle-accurate RTL models are developed. A memory bottleneck detected at such a late stage can severely limit the available design choices or even require costly redesign. In this work, we propose a novel TLM-2.0 loosely-timed contention-aware (LT-CA) modeling style that offers high-speed simulation close to traditional loosely-timed (LT) models, yet shows the same accuracy for memory contention as low-level approximately-timed (AT) models. Thus, our proposed LT-CA modeling breaks the speed/accuracy tradeoff between regular LT and AT models and offers fast and accurate observation and visualization of memory contention. Our extensible SystemC model generator automatically produces desired TLM-1 and TLM-2.0 models from a DNN architecture description for design space exploration focusing on memory contention. We demonstrate our approach with a real-world industry-strength DNN application, GoogLeNet. The experimental results show that the proposed LT-CA modeling is 46× faster in simulation than equivalent AT models with an average error of less than 1% in simulated time. Early detection of memory contentions also suggests that local memories close to computing cores can eliminate memory contention in such applications.

(Invited) Towards High-Performance Domain-Specific in-Sensor and in-Memory Accelerators

Internet of Things (IoT) devices are projected to attain an $1100B market by 2025, with a web of interconnection projected to comprise approximately 75+ billion IoT devices. The large number of IoTs consist of sensory systems that enable massive data collection from the environment and people. However, considerable portions of the captured sensory data are redundant and unstructured. Data conversion of such large raw data, storing in volatile memories, transmission, and computation in on-/off-chip processors, impose high energy consumption, latency, and a memory bottleneck at the edge. Therefore, high-speed, low-power, and normally-off computing domain-specific architectures should be explored and developed to overcome these issues. Motivated by the aforementioned concerns, we will be focusing on cross-layer (device/circuit/architecture/application) co-design of energy-efficient and high-performance processing-in-sensor and processing-in-memory platforms for implementing complex AI and machine learning tasks, bioinformatics tasks, graph processing, etc. We explain how to leverage innovations from circuits and architecture to integrate sensors, memory, and logic to break the existing memory and power walls.

Efficient Processing-in-Memory System Based on RISC-V Instruction Set Architecture

A lot of research on deep learning and big data has led to efficient methods for processing large volumes of data and research on conserving computing resources. Particularly in domains like the IoT (Internet of Things), where the computing power is constrained, efficiently processing large volumes of data to conserve resources is crucial. The processing-in-memory (PIM) architecture was introduced as a method for efficient large-scale data processing. However, PIM focuses on changes within the memory itself rather than addressing the needs of low-cost solutions such as the IoT. This paper proposes a new approach using the PIM architecture to overcome memory bottlenecks effectively in domains with computing performance constraints. We adopt the RISC-V instruction set architecture for our proposed PIM system’s design, implementation, and comprehensive performance evaluation. Our proposal expects to efficiently utilize low-spec systems like the IoT by minimizing core modifications and introducing PIM instructions at the ISA level to enable solutions that leverage PIM capabilities. We evaluate the performance of our proposed architecture by comparing it with existing structures using convolution operations, the fundamental unit of deep-learning and big data computations. The experimental results show our proposed structure achieves a 34.4% improvement in processing speed and 18% improvement in power consumption compared to conventional von Neumann-based architectures. This substantiates its effectiveness at the application level, extending to fields such as deep learning and big data.

Re2Pair: Increasing the Scalability of RePair by Decreasing Memory Usage.

The RePair compression algorithm produces a context-free grammar by iteratively substituting the most frequently occurring pair of consecutive symbols with a new symbol until all consecutive pairs of symbols appear only once in the compressed text. It is widely used in the settings of bioinformatics, machine learning, and information retrieval where random access to the original input text is needed. For example, in pangenomics, RePair is used for random access to a population of genomes. BigRePair improves the scalability of the original RePair algorithm by using Prefix-Free Parsing (PFP) to preprocess the text prior to building the RePair grammar. Despite the efficiency of PFP on repetitive text, there is a scalability issue with the size of the parse which causes a memory bottleneck in BigRePair. In this paper, we design and implement recursive RePair (denoted as ), which builds the RePair grammar using recursive PFP. Our novel algorithm faces the challenge of constructing the RePair grammar without direct access to the parse of text, relying solely on the dictionary of the text and the parse and dictionary of the parse of the text. We compare to BigRePair using SARS-CoV-2 haplotypes and haplotypes from the 1000 Genomes Project. We show that our method achieves over a 40% peak memory reduction and a speed up ranging between 12% to 79% compared to BigRePair when compressing the largest input texts in all experiments. is made publicly available under the GNU public license here: https://github.com/jkim210/Recursive-RePair.

Tucker Tensor Approach for Accelerating Fock Exchange Computations in a Real-Space Finite-Element Discretization of Generalized Kohn-Sham Density Functional Theory.

The evaluation of Fock exchange is often the computationally most expensive part of hybrid functional density functional theory calculations in a systematically improvable, complete basis. In this work, we employ a Tucker tensor based approach that substantially accelerates the evaluation of the action of Fock exchange by transforming three-dimensional convolutional integrals into a tensor product of one-dimensional convolution integrals. Our numerical implementation uses a parallelization strategy that balances the memory and communication bottlenecks, alongside overlapping compute and communication operations to enhance computational efficiency and parallel scalability. The accuracy and computational efficiency are demonstrated on various systems, including Pt clusters of various sizes and a TiO2 cluster with 3684 electrons.

DTAAD: Dual Tcn-attention networks for anomaly detection in multivariate time series data

Anomaly detection techniques enable effective anomaly detection and diagnosis in multi-variate time series data, which are of major significance for today’s industrial applications. However, establishing an anomaly detection system that can be rapidly and accurately located is a challenging problem due to the lack of anomaly labels, the high dimensional complexity of the data, memory bottlenecks in actual hardware, and the need for fast reasoning. In this paper, we propose an anomaly detection and diagnosis model, DTAAD, based on Transformer and Dual Temporal Convolutional Network (TCN). Our overall model is an integrated design in which an autoregressive model (AR) combines with an autoencoder (AE) structure. Scaling methods and feedback mechanisms are introduced to improve prediction accuracy and expand correlation differences. Constructed by us, the Dual TCN-Attention Network (DTA) uses only a single layer of Transformer encoder in our baseline experiment, belonging to an ultra-lightweight model. Our extensive experiments on seven public datasets validate that DTAAD exceeds the majority of currently advanced baseline methods in both detection and diagnostic performance. Specifically, DTAAD improved F1 scores by 8.38% and reduced training time by 99% compared to the baseline. The code and training scripts are publicly available on GitHub at https://github.com/Yu-Lingrui/DTAAD.

Programmable Racetrack for Magnetic Domain Wall Motion via Local Tuning of Exchange Biased Field

AbstractTo overcome memory bottleneck issues in memory‐centric chip technologies, in‐memory computing has been considered an alternative route involving simultaneous data storage and computing in a memory network. In the context of spintronics, a memory‐in‐logic device based on spin‐transfer‐torque or spin‐orbit‐torque magnetoresistive random‐access memory, and a magnetic domain wall (DW) racetrack has been studied. To expand the functionalities of a conventional magnetic DW racetrack, the study devises a reprogrammable exchange‐biased DW racetrack with local engineering of the exchange bias field (HE) in continuous magnetic films without requiring a lithography process for specific patterning of the films. Furthermore, current‐driven and field‐driven DW motion along the exchange‐biased racetrack is demonstrated. Additionally, within the route of the locally different exchange‐biased racetrack, a gate function can be performed to guide or stop DW motion by locally tuning HE. The complex maze racetrack is rewritable, and multiple input channels can be controlled.

Beyond von Neumann Architecture: Brain‐Inspired Artificial Neuromorphic Devices and Integrated Computing

AbstractBrain‐inspired parallel computing is increasingly considered a solution to overcome memory bottlenecks, driven by the surge in data volume. Extensive research has focused on developing memristor arrays, energy‐efficient computing strategies, and varied operational mechanisms for synaptic devices to enable this. However, to realize truly biologically plausible neuromorphic computing, it is essential to consider temporal and spatial aspects of input signals, particularly for systems based on the leaky integrate‐and‐fire model. This review highlights the significance of neuromorphic computing and outlines the fundamental components of hardware‐based neural networks. Traditionally, neuromorphic computing has relied on two‐terminal devices such as artificial synapses. However, these suffer from significant drawbacks, such as current leakage and the lack of a third terminal for precise synaptic weight adjustment. As alternatives, three‐terminal synaptic devices, including memtransistors, ferroelectric, floating‐gate, and charge‐trapped synaptic devices, as well as optoelectronic options, are explored. For an accurate replication of biological neural networks, it is vital to integrate artificial neurons and synapses, implement neurobiological functions in hardware, and develop sensory neuromorphic computing systems. This study delves into the operational mechanisms of these artificial components and discusses the integration process necessary for realizing biologically plausible neuromorphic computing, paving the way for future brain‐inspired electronic systems.

Investigating techniques to optimize data movement and reduce memory-related bottlenecks

In the ever-changing realm of computing, the importance of efficient data movement and the reduction of memory-related bottlenecks cannot be overstated. This research paper delves into a thorough examination of diverse methodologies and approaches aimed at optimizing data transfer and mitigating the constraints imposed by memory limitations. It offers an all-encompassing survey of pertinent literature, delving deep into techniques designed to enhance data movement efficiency, discussing effective strategies for alleviating memory bottlenecks, and presenting the outcomes of extensive experiments conducted. The findings of this study underscore the critical role played by these techniques in augmenting the performance, efficiency, and scalability of contemporary computing systems. In a world where the demand for computational power continues to grow, the ability to streamline data movement and overcome memory constraints is essential. By shedding light on these pivotal aspects of computing, this paper contributes to a more profound understanding of how to harness the full potential of modern computing systems, ultimately paving the way for groundbreaking advancements in the field.

Field-free spin–orbit torque switching of magnetic tunnel junction structure based on two-dimensional van der Waals WTe2

In recent years, two-dimensional van der Waals (2D vdWs) heterostructures have attracted great research interest due to their great potential in fundamental physics research and spintronic devices (such as MTJs). Due to its excellent scalability, controllable magnetism and out-of-plane anisotropy, the compact nonvolatile memory controller (NV-MC) based on spintronics is expected to solve the memory bottle-neck problem. At present, a series of in-depth studies have been conducted on advanced 2D vdWs materials, such as MoS2, WSe2, and Fe3GeTe2 (FGT). The results show that the 2D vdWs materials have great TMR value and high SOT switching efficiency, both theoretically reported and experimentally verified. In the paper, a novel MTJ device based on the FGT/WTe2 heterostructure is proposed. In the absence of an external magnetic field, the magnetization direction of the MTJ free layer can still be reversed with certainty when the unipolar write current reaches about 5 mA. Moreover, the DMI effect generated between 2-D material/FM interfaces is also considered, which can promote the performance of SOT-MTJ without the external field. The reading reliability of SOT-MRAM is improved in comparison with the traditional CoFeB-based MTJ device.

PReFormer: A memory-efficient transformer for point cloud semantic segmentation

The success of transformer networks in the natural language processing and 2D vision domains has encouraged the adaptation of transformers to 3D computer vision tasks. However, most of the existing approaches employ standard backpropagation (SBP). SBP requires the storage of model activations on a forward pass for use during the backward pass, making their memory complexity linearly proportional to model depth, hence, inefficient. Furthermore, most 3D point transformers use the classic QKV matrix multiplication design which comes with a memory bottleneck. To address these issues, we propose a memory-efficient point transformer that makes use of reversible functions and linearized self-attention to minimize SBP and transformer memory complexities, respectively. Additionally, rather than the usual UNet architectural design for segmentation, we adopt a ∇-shaped design to capture multi-size/resolution feature representations towards a finely detailed segmentation. Experimental results on benchmark datasets (Toronto3D, DALES, and CSPC) from different sensor platforms (vehicle, aerial, and backpack) show that our approach uses less than half the number of model parameters compared to its SBP counterpart. It also takes more than twice the input sequence and uses less than half the memory compared to most of the traditional approaches. Additionally, our use of a ∇-shaped architectural design yielded about a 5% increment in model performance over the U-shaped approach. Overall, the proposed PReFormer attained competitive performance compared to the state-of-the-art.

SPIMulator: A Spintronic Processing-In-Memory Simulator for Racetracks

In-memory processing is becoming a popular method to alleviate the memory bottleneck of the von Neumann computing model. With the goal of improving both latency and energy cost associated with such in-memory processing, emerging non-volatile memory technologies, such as Spintronic magnetic memory, are of particular interest as they can provide a near-SRAM read/write performance and eliminate nearly all static energy without experiencing any endurance limitations. Spintronic Racetrack Memory (RM) further addresses density concerns of spin-transfer torque memory (STT-MRAM). Moreover, it has recently been demonstrated that portions of RM nanowires can function as a polymorphic gate, which can be leveraged to implement multi-operand bulk bitwise operations. With more complex control, they can also be leveraged to build arithmetic integer and floating point processing in memory (PIM) primitives. This paper proposes SPIMulator, a Spintronic PIM sim ulator that can simulate the storage and PIM architecture of executing PIM commands in Racetrack memory. SPIMulator functionally models the polymorphic gate properties recently proposed for Racetrack memory, which allows transverse access that determines the number of ‘1’s in a segment of each Racetrack nanowire. From this simulation, SPIMulator can report real-time performance statistics such as cycle count and energy. Thus, SPIMulator simulates the multi-operand bit-wise logic operations recently proposed and can be easily extended to implement new PIM operations as they are developed. Due to the functional nature of SPIMulator, it can serve as a programming environment that allows development of PIM-based codes for verification of new acceleration algorithms. We demonstrate the value of SPIMulator through the modeling and estimations of performance and energy consumption of a variety of example applications, including the Advanced Encryption Standard (AES) for encryption primarily based on logical and look-up operations; multiplication of matrices, a frequent requirement in scientific, signal processing, and machine learning algorithms; and bitmap indices a common search table employed for database lookups.

Towards Energy-Efficient Spiking Neural Networks: A Robust Hybrid CMOS-Memristive Accelerator

Spiking Neural Networks (SNNs) are energy-efficient artificial neural network models that can carry out data-intensive applications. Energy consumption, latency, and memory bottleneck are some of the major issues that arise in machine learning applications due to their data-demanding nature. Memristor-enabled Computing-In-Memory (CIM) architectures have been able to tackle the memory wall issue, eliminating the energy and time-consuming movement of data. In this work we develop a scalable CIM-based SNN architecture with our fabricated two-layer memristor crossbar array. In addition to having an enhanced heat dissipation capability, our memristor exhibits substantial enhancement of 10% to 66% in design area, power and latency compared to state-of-the-art memristors. This design incorporates an inter-spike interval (ISI) encoding scheme due to its high information density to convert the incoming input signals into spikes. Furthermore, we include a time-to-first-spike (TTFS) based output processing stage for its energy-efficiency to carry out the final classification. With the combination of ISI, CIM and TTFS, this network has a competitive inference speed of 2μs/image and can successfully classify handwritten digits with 2.9mW of power and 2.51pJ energy per spike. The proposed architecture with the ISI encoding scheme can achieve ∼10% higher accuracy than those of other encoding schemes in the MNIST dataset.

Brain-inspired computing: can 2D materials bridge the gap between biological and artificial neural networks?

The development of excellent non-volatile storage and computing devices based on two-dimensional layered materials is necessary for overcoming the memory bottleneck of the traditional von-Neumann structure-based devices.