Scratchpad Memory Research Articles

DESCNet: Developing Efficient Scratchpad Memories for Capsule Network Hardware

Deep Neural Networks (DNNs) have been established as the state-of-the-art algorithm for advanced machine learning applications. Recently proposed by the Google Brain's team, the Capsule Networks (CapsNets) have improved the generalization ability, as compared to DNNs, due to their multi-dimensional capsules and preserving the spatial relationship between different objects. However, they pose significantly high computational and memory requirements, making their energy-efficient inference a challenging task. This paper provides, for the first time, an in-depth analysis to highlight the design and management related challenges for the (on-chip) memories deployed in hardware accelerators executing fast CapsNets inference. To enable an efficient design, we propose an application-specific memory hierarchy, which minimizes the off-chip memory accesses, while efficiently feeding the data to the hardware accelerator. We analyze the corresponding on-chip memory requirements and leverage it to propose a novel methodology to explore different scratchpad memory designs and their energy/area trade-offs. Afterwards, an application-specific power-gating technique is proposed to further reduce the energy consumption, depending upon the utilization across different operations of the CapsNets. Our results for a selected Pareto-optimal solution demonstrate no performance loss and an energy reduction of 79% for the complete accelerator, including computational units and memories, when compared to a state-of-the-art design executing Google's CapsNet model for the MNIST dataset.

SPX64

General-purpose computing systems employ memory hierarchies to provide the appearance of a single large, fast, coherent memory. In special-purpose CPUs, programmers manually manage distinct, non-coherent scratchpad memories. In this article, we combine these mechanisms by adding a virtually addressed, set-associative scratchpad to a general purpose CPU. Our scratchpad exists alongside a traditional cache and is able to avoid many of the programming challenges associated with traditional scratchpads without sacrificing generality (e.g., virtualization). Furthermore, our design delivers increased security and improves performance, especially for workloads with high locality or that interact with nonvolatile memory.

Parallelization and Optimization of NSGA-II on Sunway TaihuLight System

Sunway TaihuLight system is the first supercomputer offering a peak performance over 100 PFlops, which can be utilized to parallelize Non-dominated Sorting Genetic Algorithm II (NSGA-II), a standard approach to multi-objective optimization. However, insufficient off-chip memory bandwidth and limited scratchpad memory capacity of the supercomputer hinder the performance improvement of parallellizing NSGA-II. In this article, we propose an optimized parallel NSGA-II on Sunway TaihuLight system, called swNSGA-II, by utilizing process- and thread-level parallelism of the system based on an improved island/master-slave model. To overcome the hurdles of low memory bandwidth and capacity, we propose a data sharing scheme based on register-level communication that can efficiently parallelize non-dominated sorting and crowding-distance computation of NSGA-II. Several optimization techniques including vectorization, direct memory accessing, and double buffering are also adopted to further accelerate swNSGA-II. Experiment results show that the proposed swNSGA-II can achieve a speedup of 41284 on a use case of path planning, and a speedup of 62692 on ZDT1 as compared to conventional NSGA-II.

Space‐address decoupled scratchpad memory management for neural network accelerators

SummaryDeep neural networks have been demonstrated to be useful in varieties of intelligent tasks, and various specialized NN accelerators have been proposed recently to improve the hardware efficiency, which are typically equipped with software‐managed scratchpad memory (SPM) for high performance and energy efficiency. However, traditional SPM management techniques cause memory fragmentation for NN accelerators, and thus lead to low utilization of precious SPM. The main reason is that traditional techniques are originally designed for managing fixed‐length registers rather than variable‐length memory blocks. In this article, we propose a novel SPM management approach for NN accelerators. The basic intuition is that NN computation/memory behaviors are predictable and relatively regular compared with traditional applications, and thus most information can be determined at compile time. In addition, by exploiting the variable‐length feature of SPM, we propose to divide the allocation process into two passes: the space assignment and the address assignment pass, which are simultaneously (and implicitly) performed in traditional one‐pass allocation techniques. Experimental results on the memory requests of a representative NN accelerator demonstrate that the proposed approach can significantly reduce the memory consumption by 30% at most compared with state‐of‐the‐art SPM management techniques, and the memory usage is only 2% larger than that of the theoretical optimal allocation.

A Vector-Length Agnostic Compiler for the Connex-S Accelerator with Scratchpad Memory

Compiling sequential C programs for Connex-S, a competitive, scalable and customizable, wide vector accelerator for intensive embedded applications with 32 to 4,096 16-bit integer lanes and a limited capacity local scratchpad memory, is challenging. Our compiler toolchain uses the LLVM framework and targets OPINCAA, a JIT vector assembler and coordination C++ library for Connex-S accelerating computations for an arbitrary CPU. Therefore, we address in the compiler middle end aspects of efficient vectorization, communication, and synchronization. We perform quantitative static analysis of the program useful, among others, for the symbolic-size compiler memory allocator and the coordination mechanism of OPINCAA. We also discuss the LLVM back end for the Connex-S processor and the methodology to automatically generate instruction selection code for emulating efficiently arithmetic and logical operations for non-native types such as 32-bit integer and 16-bit floating-point. By using JIT vector assembling and by encoding the vector length of Connex-S as a parameter in the generated OPINCAA program, we achieve vector-length agnosticism to support execution on distinct embedded devices, such as several digital cameras with different resolutions, each equipped with custom-width Connex-S accelerators meant to save energy for the image processing kernels. Since Connex-S has a limited capacity local scratchpad memory of 256 KB normally, we present how we also use the PPCG C-to-C code generator to perform data tiling to minimize the total kernel execution time, subject to fitting larger program data in the local memory. We devise an accurate cost model for the Connex-S accelerator to choose optimal performance tile sizes at compile time. We successfully compile several simple benchmarks frequently used, for example, in high-performance and computer vision embedded applications. We report speedup factors of up to 11.33 when running them on a Connex-S accelerator with 128 16-bit integer lanes w.r.t. the dual-core ARM Cortex A9 host clocked at a frequency 6.67 times higher, with a total of two 128-bit Neon SIMD units.

SPECTRUM

Wireless communication standards such as Long-term Evolution (LTE) are rapidly changing to support the high data-rate of wireless devices. The physical layer baseband processing has strict real-time deadlines, especially in the next-generation applications enabled by the 5G standard. Existing basestation transceivers utilize customized DSP cores or fixed-function hardware accelerators for physical layer baseband processing. However, these approaches incur significant non-recurring engineering costs and are inflexible to newer standards or updates. Software-programmable processors offer more adaptability. However, it is challenging to sustain guaranteed worst-case latency and throughput at reasonably low-power on shared-memory many-core architectures featuring inherently unpredictable design choices, such as caches and Network-on-chip (NoC). We propose SPECTRUM , a predictable, software-defined many-core architecture that exploits the massive parallelism of the LTE/5G baseband processing workload. The focus is on designing scalable lightweight hardware that can be programmed and defined by sophisticated software mechanisms. SPECTRUM employs hundreds of lightweight in-order cores augmented with custom instructions that provide predictable timing, a purely software-scheduled NoC that orchestrates the communication to avoid any contention, and per-core software-controlled scratchpad memory with deterministic access latency. Compared to many-core architecture like Skylake-SP (average power 215 W) that drops 14% packets at high-traffic load, 256-core SPECTRUM by definition has zero packet drop rate at significantly lower average power of 24 W. SPECTRUM consumes 2.11× lower power than C66x DSP cores+accelerator platform in baseband processing. We also enable SPECTRUM to handle dynamic workloads with multiple service categories present in 5G mobile network (Enhanced Mobile Broadband (eMBB), Ultra-reliable and Low-latency Communications (URLLC), and Massive Machine Type Communications (mMTC)), using a run-time scheduling and mapping algorithm. Experimental evaluations show that our algorithm performs task/NoC mapping at run-time on fewer cores compared to the static mapping (that reserves cores exclusively for each service category) while still meeting the differentiated latency and reliability requirements.

NVM-Shelf: Secure Hybrid Encryption with Less Flip for Non-Volatile Memory

The Non-Volatile Memory (NVM), such as PRAM or STT-MRAM, is often adopted as the main memory in portable embedded systems. The non-volatility triggers a security issue against physical attacks, which is a vulnerability caused by memory extraction and snapshots. However, simply encrypting the NVM degrades the performance of the memory (high energy consumption, short lifetime), since typical encryption causes an avalanche effect while most NVMs suffer from the memory-write operation. In this paper, we propose NVM-shelf: Secure Hybrid Encryption with Less Flip (shelf) for Non-Volatile Memory (NVM), which is hybrid encryption to reduce the flip penalty. The main idea is that a stream cipher, such as block cipher CTR mode, is flip-tolerant when the keystream is reused. By modifying the CTR mode in AES block cipher, we let the keystream updated in a short period and reuse the keystream to achieve flip reduction while maintaining security against physical attacks. Since the CTR mode requires additional storage for the nonce, we classify write-intensive cache blocks and apply our CTR mode to the write-intensive blocks and apply the ECB mode for the rest of the blocks. To extend the cache-based NVM-shelf implementation toward SPM-based systems, we also propose an efficient compiler for SA-SPM: Security-Aware Scratch Pad Memory, which ensures the security of main memories in SPM-based embedded systems. Our compiler is the first approach to support full encryption of memory regions (i.e., stack, heap, code, and static variables) in an SPM-based system. By integrating the NVM-shelf framework to the SA-SPM compiler, we obtain the NVM-shelf implementation for both cache-based and SPM-based systems. The cache-based experiment shows that the NVM-shelf achieves encryption flip penalty less than 3%, and the SPM-based experiment shows that the NVM-shelf reduces the flip penalty by 31.8% compared to the whole encryption.

Cache Leakage Reduction Techniques for Hybrid SPM-Cache Architectures

Leakage energy has become an increasingly large fraction of total energy consumption, making it important to reduce leakage energy for improving the overall energy efficiency for embedded processors. In this paper, we explore how to reduce the cache leakage energy efficiently in a hybrid Scratch-Pad Memory (SPM) and cache architecture. Different from stand-alone cache, since the frequently used data may be allocated to the SPM for rapid retrieval in the hybrid architecture, the access frequency to the cache is reduced. It is possible to place the cache lines of the hybrid SPM-cache into the low power mode more aggressively than traditional leakage management for regular caches, which can reduce more leakage energy without significant performance degradation. Also, we propose a Hybrid Drowsy-Gated VDD (HDG) technique, which can adaptively exploit both short and long idle intervals of cache accesses to minimize leakage energy with insignificant performance overhead. In addition, we discussed the impact of cache size on the idle intervals of accesses, which will affect the efficiency of leakage management methods that exploit the idle intervals to reduce leakage energy.

Protecting scratchpad memory addresses against soft errors

Scratchpad memories (SPMs) are intensively utilized in modern embedded processors where a demand on improving reliability without compromising performance-predictability and area/power efficiency is on rise. Our observations imply that one target for this reliability improvement is to combat ever increasing threats of multiple bit upsets (MBUs) occurring in the SPM addresses as well as multiple event transients (METs) happening in the SPM address decoders (SADs). This paper proposes two masking techniques that protect SPM addresses in their lifespan across multiple on-chip structures prone to MBU/MET errors. The strategy behind both techniques is to replicate narrow-width SPM addresses at compile time, and to mask errors at the SAD stage. The first technique is triplication of SPM addresses (TSA), and using a triplicated SAD. For the processors having limited memory address widths, the second technique uses the concept of redundant residue number system to provide a residual replication of SPM addresses (RRSA), and a residually replicated SAD. Simulation results showed that interleaving the retained bits between replicas/residues extends error masking coverage of TSA and RRSA from 86% and 78% to 91% and 97% with no performance loss, while imposing only 0.23% and 0.11% power and 0.02% and 0.01% area to the processor.

Enhancing Matrix Multiplication With a Monolithic 3-D-Based Scratchpad Memory

Convolutional neural networks (CNNs) are one of the most popular machine learning algorithms. The convolutional layers, which account for the most execution time of CNNs, are implemented with matrix multiplication because the convolution operation performs dot products between filters and local regions of the input. On the other hand, GPUs with thousands of cores were proven to significantly accelerate matrix multiplication, compared to CPUs with a limited number of cores, especially for large matrices. However, the current memory architecture allows only one row access at a time so that multiple accesses are necessary to read the column data of the second matrix, thus slowing down matrix multiplication. In this study, we adopt the monolithic 3-D integration for the GPU scratchpad memory, called monolithic 3-D integration (M3D) scratchpad memory (SPM), to enhance matrix multiplication. The M3D SPM allows one access to read the column data of the second matrix, similar to the case of the first matrix. The simulation results show that our M3D SPM improves the system performance by 46.3% for the 32×32 matrix multiplication, over the conventional 2-D SPM where the column data of the second matrix are read sequentially.

Branch-aware data variable allocation for energy optimization of hybrid SRAM+NVM SPM☆

Abstract In this paper, we are interested in the energy optimization of hybrid Scratchpad Memory (SPM) which consists of SRAM and Non-Volatile Memory (NVM). We approach the energy optimization by data allocation with the consideration of the impact of program structure like branches. Our method is composed of two stages, i.e., program analysis stage and data allocation stage. In program analysis stage, we design two strategies to predict the branch execution of the program and reduce the number of reading/writing operations. Specifically, we propose Branch-based Static Analysis (BSA) strategy for simple programs and Neural Network Branch Prediction (NNBP) strategy for complex programs separately. In data allocation stage, we propose an Energy-based Data Allocation (EDA) strategy to reduce the energy consumption, which can determine how to allocate/migrate data variables onto SRAM/NVM. Incorporating these two stages, we have two branch-aware data variable allocation approaches, i.e., BSA+EDA for simple programs and NNBP+EDA for complex programs. Based on the existing benchmarks, we conduct extensive experiments to evaluate the proposed approaches. The experimental results show that both BSA+EDA and NNBP+EDA can effectively reduce the energy consumption up to 39.4% compared with the other approaches. Moreover, experiments demonstrate that BSA+EDA is suitable for simple programs whereas NNBP+EDA is suitable for complex programs.

Efficient parallelization of multilevel fast multipole algorithm for electromagnetic simulation on many-core SW26010 processor

A many-core parallel approach of the multilevel fast multipole algorithm (MLFMA) based on the Athread parallel programming model is presented on the homegrown many-core SW26010 CPU of China. In the proposed many-core implementation of MLFMA, the data access efficiency is improved by using data structures based on the structure of array. The adaptive workload distribution strategies are adopted on different MLFMA tree levels to ensure full utilization of computing capability and the scratchpad memory. A double buffering scheme is specially designed to make communication overlapped computation. The resulting Athread-based many-core implementation of the MLFMA is capable of solving real-life problems with over one million unknowns with a remarkable speedup. The capability and efficiency of the proposed method are analyzed through the examples of computing scattering by spheres and a practical aerocraft. Numerical results show that with the proposed parallel scheme, the total speedup ratios from 6.4 to 8.0 can be achieved, compared with the CPU master core.

Compiling for the Worst Case

Modern embedded hard real-time systems feature multiple tasks running on multiple processing cores. Schedulability analysis of such systems is usually performed on an abstract system level with each task being represented as a black box with fixed timing properties. If timing constraints are violated, then optimizing the system on a code-level to achieve schedulability is a tedious task. To tackle this issue, we propose an extension to the WCET-aware C Compiler framework WCC. We integrated an optimization framework based on Integer-Linear Programming into the WCC that is able to optimize a multi-core system with multiple tasks running on each core with regards to its schedulability. We evaluate the framework by providing two approaches on a schedulability aware static Scratchpad Memory (SPM) allocation: one based on Integer-Linear Programming (ILP) and one based on a genetic algorithm.

A Hardware-assisted Heartbeat Mechanism for Fault Identification in Large-scale IoT Systems

With increased inter-connectivity among disparate devices, such as Internet-of-Things (IoT) devices, including those deployed in a nation's critical infrastructure, there is a need to ensure that any failure in the deployed devices can be detected. The capability to automatically detect device failures is particularly crucial in a large-scale, complex IoT system, since it can be very time-consuming and challenging to investigate a large number of geographically-dispersed devices that are also of different makes and types. In this paper, we present a faulty-device identification technique that is designed to achieve lightweight processor-level architectural support. Specifically, a hardware-based monitoring agent is incorporated within a processor and connected to a separate monitoring program when an examination is required. By analyzing information collected by the agent, the monitoring program determines whether the device being monitored is functioning. Findings from our detailed evaluation demonstrate that the proposed approach can detect around 90% of the failures with minimal hardware overhead of approximately 5k gates. This area overhead is reasonable and amounts to 7.69% of the ARM Cortex-M4 a lightweight IoT-class processor that has a total area (excluding optional caches and scratch-pad memory) of 65k gates.

A 28-nm Coarse Grain 2D-Reconfigurable Array With Data Forwarding

To answer the ever-increasing demand for computational power, coarse-grain reconfigurable architectures (CGRAs) experience a revival. As they provide efficient, dedicated datapaths to match the dataflow graphs of a wide range of applications, CGRAs successfully manage high performance tasks. Yet, modern CGRAs access data through classical memory hierarchies or global scratchpad memories, failing to take advantage of data locality. This reveals memory bandwidth as a key bottleneck for these reconfigurable architectures. This letter introduces a heterogeneous 2-D CGRA which distributes data buffers across its computational grid. Together with a single-cycle data forwarding path, this allows the CGRA to better take advantage of data locality, in order to minimize data transfers. A 28-nm CMOS instantiation of this concept realizes the mapping and execution of a variety of compute intensive kernels, such as a real-time 512-point FFT or $5\times 5$ convolutional filter at high power efficiency. The CGRA demonstrates a peak energy efficiency of 584.9 GOPS/W during a 103-tap FIR filter, marking a $2.9\times $ improvement over state-of-the-art 2-D CGRAs.

SwHPFM: Refactoring and Optimizing the Structured Grid Fluid Mechanical Algorithm on the Sunway TaihuLight Supercomputer

Fluid mechanical simulation is a typical high-performance computing problem. Due to the development of high-precision parallel algorithms, traditional computing platforms are unable to satisfy the computing requirements of large-scale algorithms. The Sunway TaihuLight supercomputer, which uses the SW26010 processor as its computing node, provides a powerful computing performance for this purpose. In this paper, the Sunway hierarchical parallel fluid machinery (swHPFM) framework and algorithm are proposed. Using the proposed framework and algorithm, engineers can exploit the parallelism of the existing fluid mechanical algorithm and achieve a satisfactory performance on the Sunway TaihuLight. In the framework, a suitable mapping of the model and the system architecture is developed, and the computing power of the SW26010 processor is fully utilized via the scratch pad memory (SPM) access strategy and serpentine register communication. In addition, the framework is implemented and tested by the axial compressor rotor simulation algorithm on a real-world dataset with Sunway many-core processors. The results demonstrate that we can achieve a speedup of up to 8.2×, compared to the original ported version, which only uses management processing elements (MPEs), as well as a 1.3× speedup compared to an Intel Xeon E5 processor. The proposed framework is useful for the optimization of fluid mechanical algorithm programs on computing platforms with a heterogeneous many-core architecture.

Utilization-Aware Data Variable Allocation on NVM-Based SPM in Real-Time Embedded Systems

With the development of the nonvolatile memory (NVM), using NVM in the design of the cache and scratchpad memory (SPM) has been increased. This paper presents a data variable allocation (DVA) algorithm based on the genetic algorithm for NVM-based SPM to prolong the lifetime. The lifetime can be formulated indirectly as the write counts on each SPM address. Since the differences between global variables and stack variables, our optimization model has three constraints. The constraints of the central processing unit (CPU) utilization and size are used for all variables, while no-overlay constraint is only used for stack variables. To satisfy the constraints of the optimization model, we use the greedy strategy to generate the initial population which can determine whether data variables are allocated to SPM and distribute them evenly on SPM addresses. Finally, we use the Malardalen worst case executive time (WCET) benchmark to evaluate our algorithm. The experimental results show that the DVA algorithm can not only obtain close-to-optimal solutions, but also prolong the lifetime by 9.17% on average compared with SRAM-based SPM.

A Low-Power, High-Performance Speech Recognition Accelerator

Automatic Speech Recognition (ASR) is becoming increasingly ubiquitous, especially in the mobile segment. Fast and accurate ASR comes at high energy cost, not being affordable for the tiny power-budgeted mobile devices. Hardware acceleration reduces energy-consumption of ASR systems, while delivering high-performance. In this paper, we present an accelerator for large-vocabulary, speaker-independent, continuous speech-recognition. It focuses on the Viterbi search algorithm representing the main bottleneck in an ASR system. The proposed design consists of innovative techniques to improve the memory subsystem, since memory is the main bottleneck for performance and power in these accelerators’ design. It includes a prefetching scheme tailored to the needs of ASR systems that hides main memory latency for a large fraction of the memory accesses, negligibly impacting area. Additionally, we introduce a novel bandwidth-saving technique that removes off-chip memory accesses by 20 percent. Finally, we present a power saving technique that significantly reduces the leakage power of the accelerators scratchpad memories, providing between 8.5 and 29.2 percent reduction in entire power dissipation. Overall, the proposed design outperforms implementations running on the CPU by orders of magnitude, and achieves speedups between 1.7x and 5.9x for different speech decoders over a highly optimized CUDA implementation running on Geforce-GTX-980 GPU, while reducing the energy by 123-454x.

Neural network accelerator design with resistive crossbars: Opportunities and challenges

Deep neural networks (DNNs) achieve best-known accuracies in many machine learning tasks involved in image, voice, and natural language processing and are being used in an ever-increasing range of applications. However, their algorithmic benefits are accompanied by extremely high computation and storage costs, sparking intense efforts in optimizing the design of computing platforms for DNNs. Today, graphics processing units (GPUs) and specialized digital CMOS accelerators represent the state-of-the-art in DNN hardware, with near-term efforts focusing on approximate computing through reduced precision. However, the ever-increasing complexities of DNNs and the data they process have fueled an active interest in alternative hardware fabrics that can deliver the next leap in efficiency. Resistive crossbars designed using emerging nonvolatile memory technologies have emerged as a promising candidate building block for future DNN hardware fabrics since they can natively execute massively parallel vector-matrix multiplications (the dominant compute kernel in DNNs) in the analog domain within the memory arrays. Leveraging in-memory computing and dense storage, resistive-crossbar-based systems cater to both the high computation and storage demands of complex DNNs and promise energy efficiency beyond current DNN accelerators by mitigating data transfer and memory bottlenecks. However, several design challenges need to be addressed to enable their adoption. For example, the overheads of peripheral circuits (analog-to-digital converters and digital-to-analog converters) and other components (scratchpad memories and on-chip interconnect) may significantly diminish the efficiency benefits at the system level. Additionally, the analog crossbar computations are intrinsically subject to noise due to a range of device- and circuit-level nonidealities, potentially leading to lower accuracy at the application level. In this article, we highlight the prospects for designing hardware accelerators for neural networks using resistive crossbars. We also underscore the key open challenges and some possible approaches to address them.

Hybrid Scratchpad Video Memory Architecture for Energy-Efficient Parallel HEVC

A hybrid scratchpad video memory (Hy-SVM) for energy-efficient tiles-parallelized high-efficiency video coding (HEVC) is presented here. The key ideas behind the Hy-SVM include: application-specific design and management; combined multiple levels of private and shared memories that jointly exploit intra-tile and inter-tiles data reuse; scratchpad memories (SPMs) as on-chip data storage; SRAM; and STT-RAM hybrid design. We propose a design methodology for the Hy-SVM that leverages application-specific properties to properly define the SPMs parameters. The inter-tiles data reuse potential of parallel HEVC is exploited by our run-time overlap prediction scheme, which identifies the redundant memory access behavior by analyzing monitored past frames encoding. Based on the predicted overlap characteristics, the Hy-SVM integrates memory access management units to control the access dynamics to the private/shared SPM levels. Furthermore, adaptive access management units (APMUs) can strongly reduce on-chip energy consumption due to the predicted overlap formation. The experimental results demonstrate the Hy-SVM overall energy savings of 11%–64% (4-tile) and 8%–46% (8-tile) when compared with related works. From the external memory perspective, the Hy-SVM can improve data reuse, resulting in 14%–59% of off-chip energy consumption (compared with no inter-tiles data reuse scenarios). In addition, our APMU contributes by reducing on-chip energy consumption of the Hy-SVM by 58%, on average. Thus, compared with related works, the Hy-SVM presents the lowest on-chip energy consumption. Moreover, the overhead of implementing our management units insignificantly affects the performance- and energy-efficiency of the Hy-SVM.