On-chip Memory Research Articles

A 746 nW ECG Processor ASIC Based on Ternary Neural Network.

This paper presents an ultra-low power electrocardiography (ECG) processor application-specific integrated circuit (ASIC) for the real-time detection of abnormal cardiac rhythms (ACRs). The proposed ECG processor can support wearable or implantable ECG devices for long-term health monitoring. It adopts a derivative-based patient adaptive threshold approach to detect the R peaks in the PQRST complex of ECG signals. Two tiny machine learning classifiers are used for the accurate classification of ACRs. A 3-layer feed-forward ternary neural network (TNN) is designed, which classifies the QRS complex's shape, followed by the adaptive decision logics (DL). The proposed processor requires only 1 KB on-chip memory to store the parameters and ECG data required by the classifiers. The ECG processor has been implemented based on fully-customized near-threshold logic cells using thick-gate transistors in 65-nm CMOS technology. The ASIC core occupies a die area of 1.08 mm2. The measured total power consumption is 746 nW, with 0.8 V power supply at 2.5 kHz real-time operating clock. It can detect 13 abnormal cardiac rhythms with a sensitivity and specificity of 99.10% and 99.5%. The number of detectable ACR types far exceeds the other low power designs in the literature.

A high-performance, hardware-based deep learning system for disease diagnosis.

Modern deep learning schemes have shown human-level performance in the area of medical science. However, the implementation of deep learning algorithms on dedicated hardware remains a challenging task because modern algorithms and neuronal activation functions are generally not hardware-friendly and require a lot of resources. Recently, researchers have come up with some hardware-friendly activation functions that can yield high throughput and high accuracy at the same time. In this context, we propose a hardware-based neural network that can predict the presence of cancer in humans with 98.23% accuracy. This is done by making use of cost-efficient, highly accurate activation functions, Sqish and LogSQNL. Due to its inherently parallel components, the system can classify a given sample in just one clock cycle, i.e., 15.75 nanoseconds. Though this system is dedicated to cancer diagnosis, it can predict the presence of many other diseases such as those of the heart. This is because the system is reconfigurable and can be programmed to classify any sample into one of two classes. The proposed hardware system requires about 983 slice registers, 2,655 slice look-up tables, and only 1.1 kilobits of on-chip memory. The system can predict about 63.5 million cancer samples in a second and can perform about 20 giga-operations per second. The proposed system is about 5–16 times cheaper and at least four times speedier than other dedicated hardware systems using neural networks for classification tasks.

Dynamic Optimization of On-Chip Memories for HLS Targeting Many-Accelerator Platforms

Many-accelerator platforms have been introduced for maximizing FPGA's throughput. However, as the high saturation rate of the FPGA's on-chip memories limits the number of synthesized accelerators, frameworks for Dynamic Memory Management (DMM) that allow the synthesized designs to allocate/de-allocate on-chip memory resources during run-time have been suggested. Although, those frameworks manage to increase the accelerators’ density by minimizing the utilized memory resources, the parallel execution of many-accelerators may cause severe memory fragmentation and thus memory allocation failures. In this work, a framework that optimizes the memory usage by performing memory defragmentation operations in HLS many-accelerator architectures that share on-chip memories is proposed. Experimental results highlight the effectiveness of the proposed solution to eliminate memory allocation failures due to memory fragmentation, reduce memory allocation failures up to 32% on average and decrease the memory size requirements up to 5% with controllable latency and resource utilization overhead.

CRP: Conditional Replacement Policy for Reliability Enhancement of STT-MRAM Caches

Driven by the trends of emerging technologies in on-chip memories, with increasing the size of last-level caches (LLCs), spin-transfer torque magnetic random access memories (STT-MRAMs) are the most promising alternative technology among the non-volatile memories (NVMs) to replace SRAMs. Despite their high density, scalability, and near-zero leakage power, the reliability of STT-MRAM LLCs is threatened by a high error rate due to their stochastic switching behavior. The error rate is highly influenced by the cache management, which necessitates redesigning the cache replacement policy based on the error behavior. In this article, we propose an error-aware cache replacement policy, namely, conditional replacement policy (CRP), to improve the reliability of STT-MRAM caches by decreasing the rate of both read disturbance and write failure. This is ascertained by nominating an appropriate data block in the cache to be replaced with an incoming data block, considering the minimum error rate. Moreover, the performance and latency of both write and read operations are considered. The simulation results show that compared with the state-of-the-art replacement policy in STT-MRAM caches, CRP reduces the total error rate by 68%. In addition, the proposed policy enhances the performance by 2% and saves energy consumption by 19%, on average. Meanwhile, the total number of writes is decreased by 41% compared with the previous scheme.

Ascend: A Scalable and Energy-Efficient Deep Neural Network Accelerator With Photonic Interconnects

The complexity and size of recent deep neural network (DNN) models have increased significantly in pursuit of high inference accuracy. Chiplet-based accelerator is considered a viable scaling approach to provide substantial computation capability and on-chip memory for efficient process of such DNN models. However, communication using metallic interconnects in prior chiplet-based accelerators poses a major challenge to system performance, energy efficiency, and scalability. Photonic interconnects can adequately support communication across chiplets due to features such as distance-independent latency, high bandwidth density, and high energy efficiency. Furthermore, the salient ease of broadcast property makes photonic interconnects suitable for DNN inference which often incurs prevalent broadcast communication. In this paper, we propose a scalable chiplet-based DNN accelerator with photonic interconnects named ASCEND. ASCEND introduces (1) a novel photonic network that supports seamless intra- and inter- chiplet broadcast communication, and flexible mapping of diverse convolution layers, and (2) a tailored dataflow that exploits the ease of broadcast property and maximizes parallelism by simultaneously processing computations with shared input data. Simulation results using multiple DNN models show that ASCEND achieves 71% and 67% reduction in execution time and energy consumption, respectively, as compared to other state-of-the-art chiplet-based DNN accelerators with metallic or photonic interconnects.

RASHT: A Partially Reconfigurable Architecture for Efficient Implementation of CNNs

Convolutional neural networks (CNNs) are widely used in machine learning (ML) applications such as image processing. CNN requires heavy computations to provide significant accuracy for many ML tasks. Therefore, the efficient implementations of CNNs to improve performance using limited resources without accuracy reduction is a challenge for ML systems. One of the architectures for the efficient execution of CNNs is the array-based accelerator, that consists of an array of similar processing elements (PEs). The array accelerators are popular as high-performance architecture using the features of parallel computing and data reuse. These accelerators are optimized for a set of CNN layers, not for individual layers. Using the same accelerator dimension size to compute all CNN layers with varying shapes and sizes leads to the resource underutilization problem. We propose a flexible and scalable architecture for array-based accelerator that increases resource utilization by resizing PEs to better match the different shapes of CNN layers. The low-cost partial reconfiguration improves resource utilization and performance, resulting in a 23.2% reduction in computational times of GoogLeNet compared to the state-of-the-art accelerators. The proposed architecture decreases the on-chip memory access rate by 26.5% with no accuracy loss.

An integrated photonic device for on-chip magneto-optical memory reading

AbstractThis study presents the design, fabrication and experimental demonstration of a magneto-photonic device that delivers non-volatile photonic memory functionality. The aim is to overcome the energy and speed bottleneck of back-and-forth signal conversion between the electronic and optical domains when retrieving information from non-volatile memory. The device combines integrated photonic components based on the InP membrane on silicon (IMOS) platform and a non-volatile, built-in memory element (ferromagnetic thin-film multilayers) realized as a top-cladding on the photonic waveguides (a post-processing step). We present a design where the phase of the guided light is engineered via two mechanisms: the polar magneto-optical Kerr effect (MOKE) and the propagation in an asymmetrical cross-section (triangular) waveguide. Thanks to its design, the device yields different mode-specific transmissions depending on the memory state it encodes. We demonstrate the recording of the magnetic hysteresis using the transmitted optical signal, providing direct proof for all optical magnetic memory reading using an integrated photonic chip. Using mathematical model and optical simulations, we support the experimental observations and quantitatively reproduce the Kerr signal amplitudes on-chip. A 1% transmitted power contrast from devices is promising indicating that in a shot noise limited scenario the theoretical bandwidth of memory read-out exceeds 50 Gbits/s.

A novel read decoupled 8T1M nvSRAM cell for near threshold operation

The SRAM is mainstay in on-chip memories due to easier integration with standard processing technology and high performance. The SRAM cell is required to have the reliable performance of at strong inversion region as well as near threshold region. The performance of the SRAM cell degrades near threshold voltage due to the increase in the susceptibility to the noise. Also, the idle cells in the SRAM array consume power which is undesirable. A new nvSRAM cell is introduced in this paper which can maintain its performance near threshold voltage. The power consumption is reduced up to 99.7% during write ‘1’ operation. Also, the writability is improved by 69% in comparison to the existing counterparts. The store/restore delay and energy are improved by 70% and 47% respectively. The simulations are carried out using PTM model of 32 nm PTM model of CMOS technology node for different supply voltage. The simulations at different technology node are also carried out to validate the results.

MoRS: An Approximate Fault Modeling Framework for Reduced-Voltage SRAMs

On-chip memory (usually based on Static RAMs-SRAMs) are crucial components for various computing devices including heterogeneous devices, e.g, GPUs, FPGAs, ASICs to achieve high performance. Modern workloads such as Deep Neural Networks (DNNs) running on these heterogeneous fabrics are highly dependent on the on-chip memory architecture for efficient acceleration. Hence, improving the energy-efficiency of such memories directly leads to an efficient system. One of the common methods to save energy is undervolting i.e., supply voltage underscaling below the nominal level. Such systems can be safely undervolted without incurring faults down to a certain voltage limit. This safe range is also called voltage guardband. However, reducing voltage below the guardband level without decreasing frequency causes timing-based faults. In this paper, we propose MoRS, a framework that generates the first approximate undervolting fault model using real faults extracted from experimental undervolting studies on SRAMs to build the model. We inject the faults generated by MoRS into the on-chip memory of the DNN accelerator to evaluate the resilience of the system under the test. MoRS has the advantage of simplicity without any need for high-time overhead experiments while being accurate enough in comparison to a fully randomly-generated fault injection approach. We evaluate our experiment in popular DNN workloads by mapping weights to SRAMs and measure the accuracy difference between the output of the MoRS and the real data. Our results show that the maximum difference between real fault data and the output fault model of MoRS is 6.21%, whereas the maximum difference between real data and random fault injection model is 23.2%. In terms of average proximity to the real data, the output of MoRS outperforms the random fault injection approach by 3.21x.

A Heterogeneous In-Memory Computing Cluster for Flexible End-to-End Inference of Real-World Deep Neural Networks

Deployment of modern TinyML tasks on small battery-constrained IoT devices requires high computational energy efficiency. Analog In-Memory Computing (IMC) using non-volatile memory (NVM) promises major efficiency improvements in deep neural network (DNN) inference and serves as on-chip memory storage for DNN weights. However, IMC’s functional flexibility limitations and their impact on performance, energy, and area efficiency are not yet fully understood at the system level. To target practical end-to-end IoT applications, IMC arrays must be enclosed in heterogeneous programmable systems, introducing new system-level challenges which we aim at addressing in this work. We present a heterogeneous tightly-coupled clustered architecture integrating 8 RISC-V cores, an in-memory computing accelerator (IMA), and digital accelerators. We benchmark the system on a highly heterogeneous workload such as the Bottleneck layer from a MobileNetV2, showing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.5\times $ </tex-math></inline-formula> performance and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$9.5\times $ </tex-math></inline-formula> energy efficiency improvements, compared to highly optimized parallel execution on the cores. Furthermore, we explore the requirements for end-to-end inference of a full mobile-grade DNN (MobileNetV2) in terms of IMC array resources, by scaling up our heterogeneous architecture to a multi-array accelerator. Our results show that our solution, on the end-to-end inference of the MobileNetV2, is one order of magnitude better in terms of execution latency than existing programmable architectures and two orders of magnitude better than state-of-the-art heterogeneous solutions integrating in-memory computing analog cores.

Resource-constrained FPGA implementation of YOLOv2

Progress is being made to deploy convolutional neural networks (CNNs) into the Internet of Things (IoT) edge devices for handling image analysis tasks locally. These tasks require low-latency and low-power computation on low-resource IoT edge devices. However, CNN-based algorithms, e.g. YOLOv2, typically contain millions of parameters. With the increase in the CNN’s depth, filters are increased by a power of two. A large number of filters and operations could lead to frequent off-chip memory access that affects the operation speed and power consumption of the device. Therefore, it is a challenge to map a deep CNN into a low-resource edge IoT platform. To address this challenge, we present a resource-constrained Field-Programmable Gate Array implementation of YOLOv2 with optimized data transfer and computing efficiency. Firstly, a scalable cross-layer dataflow strategy is proposed which allows on-chip data transfer between different types of layers, and offers flexible off-chip data transfer when the intermediate results are unaffordable on-chip. Next, a filter-level data-reuse dataflow strategy together with a filter-level parallel multiply-accumulate operation computing processing elements array is developed. Finally, multi-level sliding buffers are developed to optimize the convolutional computing loop and reuse the input feature maps and weights. Experiment results show that our implementation has achieved 4.8 W of low-power consumption for executing YOLOv2, an 8-bit deep CNN containing 50.6 MB weights, using low-resource of 8.3 Mbits on-chip memory. The throughput and power efficiency are 100.33 GOP/s and 20.90 GOP/s/W, respectively.

CASSANN-v2: A high-performance CNN accelerator architecture with on-chip memory self-adaptive tuning

CASSANN-v2: A high-performance CNN accelerator architecture with on-chip memory self-adaptive tuning

An ASIP for Neural Network Inference on Embedded Devices with 99% PE Utilization and 100% Memory Hidden under Low Silicon Cost.

The computation efficiency and flexibility of the accelerator hinder deep neural network (DNN) implementation in embedded applications. Although there are many publications on deep neural network (DNN) processors, there is still much room for deep optimization to further improve results. Multiple dimensions must be simultaneously considered when designing a DNN processor to reach the performance limit of the architecture, including architecture decision, flexibility, energy efficiency, and silicon cost minimization. Flexibility is defined as the ability to support as many multiple networks as possible and to easily adjust the scale. For energy efficiency, there are huge opportunities for power efficiency optimization, which involves access minimization and memory latency minimization based on on-chip memory minimization. Therefore, this work focused on low-power and low-latency data access with minimized silicon cost. This research was implemented based on an ASIP (application specific instruction set processor) in which an ISA was based on the caffe2 inference operator and the hardware design was based on a single instruction multiple data (SIMD) architecture. The scalability and system performance of our SoC extension scheme were demonstrated. The VLIW was used to execute multiple instructions in parallel. All costs for data access time were thus eliminated for the convolution layer. Finally, the processor was synthesized based on TSMC 65 nm technology with a 200 MHz clock, and the Soc extension scheme was analyzed in an experimental model. Our design was tested on several typical neural networks, achieving 196 GOPS at 200 MHz and 241 GOPS/W on the VGG16Net and AlexNet.

Application-aware aging analysis and mitigation for SRAM Design-for-Relability

Most embedded systems integrate several on-chip Static Random Access Memories (SRAMs). These SRAMs are increasingly susceptible to reliability threats such as Bias Temperature Instability (BTI) due to the continuous trend of technology shrinking. BTI leads to a significant performance degradation, especially in the Sense Amplifiers (SAs) of SRAMs, where failures are fatal, since the data of a whole column is destroyed.In this paper we present a new workload-aware aging analysis for On-Chip SRAMs. The method incorporates the workload of embedded applications while considering aging in the complete read-path and its control signals. According to this workload, we predict the performance degradation in the memory and its end of lifetime. Furthermore, we present the Mitigation of AGIng Circuitry (MAGIC), a low-cost circuitry to effectively mitigate aging in SAs by wear-leveling.We show that SRAM cell and SA aging have a significant contribution to the overall degradation of the read-path while aging in the SA's control signals counter-intuitively leads to minor performance improvements. The proposed mitigation scheme MAGIC can mitigate the degradation in the read-path up to ∼26% while introducing minimal area/performance overhead. Our application-aware end-of-life analysis shows that this translates into 3x longer lifetime.

Integration of Single-Port Memory (ISPM) for Multiprecision Computation in Systolic-Array-Based Accelerators

On-chip memory is one of the core components of deep learning accelerators. In general, the area used by the on-chip memory accounts for around 30% of the total chip area. With the increasing complexity of deep learning algorithms, it will become a challenge for the accelerators to integrate much larger on-chip memory responding to algorithm needs, whereas the on-chip memory for multiprecision computation is required by the different precision (such as FP32, FP16) computations in training and inference. To solve it, this paper explores the use of single-port memory (SPM) in systolic-array-based deep learning accelerators. We propose transformation methods for multiple precision computation scenarios, respectively, to avoid the conflict of simultaneous read and write requests on the SPM. Then, we prove that the two methods are feasible and can be implemented on hardware without affecting the computation efficiency of the accelerator. Experimental results show that both methods have about 30% and 25% improvement in terms of area cost when accelerator integrates SPM without affecting the throughput of the accelerator, while the hardware cost is almost negligible.

Gated-CNN: Combating NBTI and HCI aging effects in on-chip activation memories of Convolutional Neural Network accelerators

Negative Bias Temperature Instability (NBTI) and Hot Carrier Injection (HCI) are two of the main reliability threats in current technology nodes. These aging phenomena degrade the transistor’s threshold voltage (Vth) over the lifetime of a digital circuit, resulting in slower transistors that eventually lead to a faulty operation when the critical paths become longer than the processor cycle time. Among all the transistors on a chip, the most vulnerable transistors to such wearout effects are those used to implement SRAM storage, since memory cells are continuously degrading. In particular, NBTI ages PMOS cell transistors when a given logic value is stored for a long period (i.e., a long duty cycle), whereas HCI ages NMOS cell transistors not only when the stored value flips but also when it is accessed. This work focuses on mitigating aging in the on-chip SRAM memories of Convolutional Neural Network (CNN) accelerators storing activations. This paper makes two main contributions. At the software level, we quantify the aging induced by current CNN benchmarks with a characterization study of duty cycle, flip, and access patterns in every activation memory cell. Based on the insights from this study, this work proposes a novel microarchitectural technique, Gated-CNN, that ensures a uniform aging degradation of every memory cell. To do so, Gated-CNN exploits power-gating and address rotation techniques tailored to the memory demands and temporal/spatial localities exhibited by CNN applications, as well as the memory organization and management of CNN accelerators. Experimental results show that, compared to a conventional design, the average Vth degradation savings are at least as much as 49% depending on the type of transistor.

Light-induced thermomagnetic recording of thin-film magnet CoFeB on silicon waveguide for on-chip magneto-optical memory.

Thermomagnetic recording is a technique used as a writing process for magneto-optical (MO) drives. Despite their significant advantages, such as rewritability, nonvolatility, reliability, and large cycling endurance, MO drives are rarely used today because of the complex drive systems that must deal with magnetic field and lightwave simultaneously. This study reports on the light-induced thermomagnetic recording of a ferromagnetic thin-film CoFeB on a Si photonic platform. Lightwave guided in the Si waveguide evanescently coupled to the thin-film magnet and underwent optical absorption, resulting in heating and a decrease in coercive force. Therefore, we observed magnetization reversal with an applied magnetic field for both continuous and modulated light pulses using a magneto-optical Kerr effect microscope, and the light-induced thermomagnetic recording was experimentally demonstrated on a Si photonic platform. The proposed scheme enables the realization of on-chip MO memories on the Si photonic platform in which neither bulky free-space optics nor mechanical rotation systems are required.

Progress toward picosecond on-chip magnetic memory

We offer a perspective on the prospects of ultrafast spintronics and opto-magnetism as a pathway to high-performance, energy-efficient, and non-volatile embedded memory in digital integrated circuit applications. Conventional spintronic devices, such as spin-transfer-torque magnetic-resistive random-access memory (STT-MRAM) and spin–orbit torque MRAM, are promising due to their non-volatility, energy-efficiency, and high endurance. STT-MRAMs are now entering into the commercial market; however, they are limited in write speed to the nanosecond timescale. Improvement in the write speed of spintronic devices can significantly increase their usefulness as viable alternatives to the existing CMOS-based devices. In this article, we discuss recent studies that advance the field of ultrafast spintronics and opto-magnetism. An optimized ferromagnet–ferrimagnet exchange-coupled magnetic stack, which can serve as the free layer of a magnetic tunnel junction (MTJ), can be optically switched in as fast as ∼3 ps. Integration of ultrafast magnetic switching of a similar stack into an MTJ device has enabled electrical readout of the switched state using a relatively larger tunneling magnetoresistance ratio. Purely electronic ultrafast spin–orbit torque induced switching of a ferromagnet has been demonstrated using ∼6 ps long charge current pulses. We conclude our Perspective by discussing some of the challenges that remain to be addressed to accelerate ultrafast spintronics technologies toward practical implementation in high-performance digital information processing systems.

A Highly Miniaturized, Chronically Implanted ASIC for Electrical Nerve Stimulation.

We present a wireless, fully implantable device for electrical stimulation of peripheral nerves consisting of a powering coil, a tuning network, a Zener diode, selectable stimulation parameters, and a stimulator IC, all encapsulated in biocompatible silicone. A wireless RF signal at 13.56 MHz powers the implant through the on-chip rectifier. The ASIC, designed in TSMC's 180 nm MS RF G process, occupies an area of less than 1.2 mm2. The IC enables externally selectable current-controlled stimulation through an on-chip read-only memory with a wide range of 32 stimulation parameters (90-750 µA amplitude, 100 µs or 1 ms pulse width, 15 or 50 Hz frequency). The IC generates the constant current waveform using an 8-bit binary weighted DAC and an H-Bridge. At the most power-hungry stimulation parameter, the average power consumption during a stimulus pulse is 2.6 mW with a power transfer efficiency of ∼5.2%. In addition to benchtop and acute testing, we chronically implanted two versions of the device (a design with leads and a leadless design) on two rats' sciatic nerves to verify the long-term efficacy of the IC and the full system. The leadless device had the following dimensions: height of 0.45 cm, major axis of 1.85 cm, and minor axis of 1.34 cm, with similar dimensions for the device with leads. Both devices were implanted and worked for experiments lasting from 21-90 days. To the best of our knowledge, the fabricated IC is the smallest constant-current stimulator that has been tested chronically.

A reconfigurable processor for mix-precision CNNs on FPGA

To solve the problem of low computing efficiency of existing accelerators for convolutional neural network (CNNs), which caused by the inability to adapt to the characteristics of computing mode and caching for the mixed-precision quantized CNNs model, we propose a reconfigurable CNN processor in this paper, which consists of the reconfigurable adaptable computing unit, flexible on-chip cache unit and macro-instruction set. The multi-core CNN processor can be reconstructed according to the structure of CNN models and constraints of reconfigurable resources, to improve the utilization of computing resources. The elastic on-chip buffer and the data access approach by dynamically configuring an address to better utilization of on-chip memory. Then, the macroinstruction set architecture (mISA) can fully express the characteristics of the mixed-precision CNN models and reconfigurable processors, to reduce the complexity of mapping CNNs with different network structures and computing modes to reconfigurable the CNNs processors. For the well-known CNNs-VGG16 and ResNet-50, the proposed CNN processor has been implemented using Ultra96-V2 and ZCU102 FPGA, showing the throughput of 216.6 GOPS, and 214 GOPS, the computing efficiency of 0.63 GOPS/DSP and 0.64 GOPS/DSP on Ultra96-V2, respectively, achieving a better efficiency than the CNN accelerator based on fixed bit-width. Meanwhile, for ResNet-50, the throughput and the computing efficiency are up to 931.8 GOPS, 0.40 GOPS/DSP on ZCU102, respectively. In addition, these achieve up to 55.4% higher throughput than state-of-the-art CNN accelerators.