Memory Access Operations Research Articles

A Sliding-Kernel Computation-In-Memory Architecture for Convolutional Neural Network.

Presently described is a sliding-kernel computation-in-memory (SKCIM) architecture conceptually involving two overlapping layers of functional arrays, one containing memory elements and artificial synapses for neuromorphic computation, the other is used for storing and sliding convolutional kernel matrices. A low-temperature metal-oxide thin-film transistor (TFT) technology capable of monolithically integrating single-gate TFTs, dual-gate TFTs, and memory capacitors is deployed for the construction of a physical SKCIM system. Exhibiting an 88% reduction in memory access operations compared to state-of-the-art systems, a 32 × 32 SKCIM system is applied to execute common convolution tasks. A more involved demonstration is the application of a 5-layer, SKCIM-based convolutional neural network to the classification of the modified national institute of standards and technology (MNIST) dataset of handwritten numerals, achieving an accuracy rate of over 95%.

A Hardware Non-Invasive Mapping Method for Condition Bits in Binary Translation

Binary translation, as an important bridge for application compatibility between different instruction set architectures (ISAs), has attracted much attention in the industry. However, due to hardware resource limitations of the target ISA, the translation efficiency and the practicability are poor. Recently, Apple has made it possible to run x86 programs on ARM through a translation technology called Rosetta based on software-hardware collaboration. In this paper, we proposed a hardware non-invasive mapping method for condition bits (HNIMCB) in binary translation, which innovatively implements the setting and referencing operations of the condition bits without changing the original instruction encoding and function of the target processor. This method is applicable for binary translation from source architectures with condition bit operations to target architectures without condition bit operations. It eliminates the difference of conditional bit resources between the source and target ISAs, reduces the computational instructions and memory access operations after translation from the source to the target ISA, and dramatically improves the translation efficiency. We conducted this experiment on a functional simulation level using the QEMU binary translator from ARM to RISC-V. A series of benchmark tests revealed that the total number of instructions decreased by 41%, while the number of memory access instructions decreased by 37% after the translation applying with the HNIMCB.

MLiM: High-Performance Magnetic Logic in-Memory Scheme With Unipolar Switching SOT-MRAM

Conventional computing architectures based on the von Neumann structure are suffering from the severe ‘memory wall’ issue due to the isolation and speed mismatch between memory and processor. As a promising solution, the concept of logic in-memory (LiM) has been proposed to effectively reduce the overhead of data migration and has been extensively studied in various memory technologies such as SRAM, DRAM, MRAM, ReRAM, etc. Among them, SOT-MRAM combines the advantages of non-volatility, low static power consumption, ultra-fast read/write speed, and high density, has emerged as one of the most promising candidates for low-power LiM implementations. In this paper, four in-memory logic operations, AND, OR, MAJ and full-addition (FA), are proposed based on the Unipolar Switching (US) SOT-MRAM devices. Incorporating the emerging switching behavior of SOT-MRAM, these operations can be performed with the basic memory access operations (read/write) with negligible modifying peripheral circuits. Meanwhile, by optimizing the operation steps, the performance degradation caused by the instability of SOT-MRAM device can be minimized in the proposed LiM architecture. Detailed simulation results show that the proposed design can reduce the latency (energy) of AND, OR operations at least by 71.2%, 74.4% (30.0%, 35.4%) compared with the existing SRAM and STT-MRAM designs. For MAJ and FA operations, the performance is improved by at least 34.7% and 44.8% compared to the existing design. The robustness of our design is demonstrated by the 100% pass of the 1000 samples Monte Carlo simulations for the sufficient switching current margin and the effectiveness of basic operations.

Efficient training of spiking neural networks with temporally-truncated local backpropagation through time.

Directly training spiking neural networks (SNNs) has remained challenging due to complex neural dynamics and intrinsic non-differentiability in firing functions. The well-known backpropagation through time (BPTT) algorithm proposed to train SNNs suffers from large memory footprint and prohibits backward and update unlocking, making it impossible to exploit the potential of locally-supervised training methods. This work proposes an efficient and direct training algorithm for SNNs that integrates a locally-supervised training method with a temporally-truncated BPTT algorithm. The proposed algorithm explores both temporal and spatial locality in BPTT and contributes to significant reduction in computational cost including GPU memory utilization, main memory access and arithmetic operations. We thoroughly explore the design space concerning temporal truncation length and local training block size and benchmark their impact on classification accuracy of different networks running different types of tasks. The results reveal that temporal truncation has a negative effect on the accuracy of classifying frame-based datasets, but leads to improvement in accuracy on event-based datasets. In spite of resulting information loss, local training is capable of alleviating overfitting. The combined effect of temporal truncation and local training can lead to the slowdown of accuracy drop and even improvement in accuracy. In addition, training deep SNNs' models such as AlexNet classifying CIFAR10-DVS dataset leads to 7.26% increase in accuracy, 89.94% reduction in GPU memory, 10.79% reduction in memory access, and 99.64% reduction in MAC operations compared to the standard end-to-end BPTT. Thus, the proposed method has shown high potential to enable fast and energy-efficient on-chip training for real-time learning at the edge.

CmnRec: Sequential Recommendations With Chunk-Accelerated Memory Network

Recently, Memory-based Neural Recommenders (MNR) have demonstrated superior predictive accuracy in the task of sequential recommendations, particularly for modeling long-term item dependencies. However, typical MNR requires complex memory access operations, i.e., both writing and reading via a controller (e.g., RNN) at every time step. Those frequent operations will dramatically increase the network training time, resulting in the difficulty in being deployed on industrial-scale recommender systems. In this paper, we present a novel general <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Chunk</i> framework to accelerate MNR significantly. Specifically, our framework divides proximal information units into chunks, and performs memory access at certain time steps, whereby the number of memory operations can be greatly reduced. We investigate two ways to implement effective chunking, i.e., PEriodic Chunk (PEC) and Time-Sensitive Chunk (TSC), to preserve and recover important recurrent signals in the sequence. Since chunk-accelerated MNR models take into account more proximal information units than that from a single timestep, it can alleviate the influence of noise in the user-item interaction sequence to a large extent, and thus improve the stability of MNR. In this way, the proposed chunk mechanism can lead to not only faster training and prediction, but even slightly better results. The experimental results on three real-world datasets (weishi, ml-10M and ml-latest) show that our chunk framework notably reduces the running time (e.g., with up to 7x for training & 10x for inference on ml-latest) of MNR, and meantime achieves competitive performance.

Unified Cross-Component Linear Model in VVC Based on a Subset of Neighboring Samples

To compress industrial video content efficiently, H.266/Versatile Video Coding (VVC) introduces cross-component linear model (CCLM) prediction as a new coding tool, in which chroma components are predicted from the luma component based on a linear model. In this article, we propose a subset-based CCLM (S-CCLM), in which the model parameters are derived based on a subset of neighboring samples. To choose the most proper subset, we build the relationship between the prediction error and the geometric distance and resolve the optimal subset construction problem by minimizing the geometric distance. With the well-designed subset, a weight-guided parameter derivation algorithm is further proposed to improve the accuracy of the model parameters. The experimental results show that the proposed S-CCLM can achieve Bjontegaard delta bitrate (BD-rate) reductions of 0.14%, 0.64%, and 0.75% for the Y, Cb, and Cr components, respectively, when the number of samples in the subset, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> , is 4 and BD-rate reductions of 0.22%, 0.80%, and 0.95% when <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> is 8. Given a small fixed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> , fewer memory access operations are needed during the CCLM calculation, and a unified CCLM process can be achieved for coding blocks with different sizes and different modes. Due to its hardware-friendly architecture, the S-CCLM has been partially adopted by H.266/VVC.

Instant neural graphics primitives with a multiresolution hash encoding

Neural graphics primitives, parameterized by fully connected neural networks, can be costly to train and evaluate. We reduce this cost with a versatile new input encoding that permits the use of a smaller network without sacrificing quality, thus significantly reducing the number of floating point and memory access operations: a small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through stochastic gradient descent. The multiresolution structure allows the network to disambiguate hash collisions, making for a simple architecture that is trivial to parallelize on modern GPUs. We leverage this parallelism by implementing the whole system using fully-fused CUDA kernels with a focus on minimizing wasted bandwidth and compute operations. We achieve a combined speedup of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds, and rendering in tens of milliseconds at a resolution of 1920×1080.

Partial sums on the ultra-wide word RAM

We consider the classic partial sums problem on the ultra-wide word RAM model of computation. This model extends the classic w-bit word RAM model with special ultrawords of length w2 bits that support standard arithmetic and boolean operation and scattered memory access operations that can access w (non-contiguous) locations in memory. The ultra-wide word RAM model captures (and idealizes) modern vector processor architectures.Our main result is a new in-place data structure for the partial sum problem that only stores a constant number of ultrawords in addition to the input and supports operations in doubly logarithmic time. This matches the best known time bounds for the problem (among polynomial space data structures) while improving the space from superlinear to a constant number of ultrawords. Our results are based on a simple and elegant in-place word RAM data structure, known as the Fenwick tree. Our main technical contribution is a new efficient parallel ultra-wide word RAM implementation of the Fenwick tree, which is likely of independent interest.

Nanopore base calling on the edge.

MotivationMinION is a portable nanopore sequencing device that can be easily operated in the field with features including monitoring of run progress and selective sequencing. To fully exploit these features, real-time base calling is required. Up to date, this has only been achieved at the cost of high computing requirements that pose limitations in terms of hardware availability in common laptops and energy consumption.ResultsWe developed a new base caller DeepNano-coral for nanopore sequencing, which is optimized to run on the Coral Edge Tensor Processing Unit, a small USB-attached hardware accelerator. To achieve this goal, we have designed new versions of two key components used in convolutional neural networks for speech recognition and base calling. In our components, we propose a new way of factorization of a full convolution into smaller operations, which decreases memory access operations, memory access being a bottleneck on this device. DeepNano-coral achieves real-time base calling during sequencing with the accuracy slightly better than the fast mode of the Guppy base caller and is extremely energy efficient, using only 10 W of power.Availability and implementation https://github.com/fmfi-compbio/coral-basecaller Supplementary information Supplementary data are available at Bioinformatics online.

Parallel algorithms for the creation of medical database

Analysis of parallel algorithms for graphics processors allows you to determine the bottlenecks of the algorithm that affect its performance on a particular computing system. Algorithms can be analyzed both at the AGM level and at the level of the CPU-GPU system. Despite the fact that the model does not reflect the overlap of the execution of computational operations and operations of access to the global memory, but estimates the upper limit of the execution time of the algorithm on the AGM, in any parallel algorithms for the AGM it is necessary to reduce this parameter of the algorithm due to the large number of clock cycles spent by the multiprocessor to implement global memory access operations.

Acceleration of vector bilateral filtering for hyperspectral imaging with GPU

SummaryFor hyperspectral imaging, the vector bilateral filter usually leads to better performance when compared with the traditional 2D bilateral filter. However, the large computation complexity of vector bilateral filtering makes it an extremely time cost algorithm. To overcome this challenge, a GPU‐based acceleration for vector bilateral filtering called vBF_GPU was proposed in this paper. To improve the efficiency of the cache memory usage, multiple CUDA threads were utilized to processing one pixel of the hyperspectral image in vBF_GPU. The memory access operation of vBF_GPU was fully optimized to reduce the memory access cost of the GPU program. The experiment results indicated that vBF_GPU can provide more than 30× speedup when compared with an octa‐core CPU implementation and more than 20× speedup when compared with a naïve GPU implementation of vector bilateral filtering.

Deep Learning on Computational-Resource-Limited Platforms: A Survey

Nowadays, Internet of Things (IoT) gives rise to a huge amount of data. IoT nodes equipped with smart sensors can immediately extract meaningful knowledge from the data through machine learning technologies. Deep learning (DL) is constantly contributing significant progress in smart sensing due to its dramatic superiorities over traditional machine learning. The promising prospect of wide-range applications puts forwards demands on the ubiquitous deployment of DL under various contexts. As a result, performing DL on mobile or embedded platforms is becoming a common requirement. Nevertheless, a typical DL application can easily exhaust an embedded or mobile device owing to a large amount of multiply and accumulate (MAC) operations and memory access operations. Consequently, it is a challenging task to bridge the gap between deep learning and resource-limited platforms. We summarize typical applications of resource-limited deep learning and point out that deep learning is an indispensable impetus of pervasive computing. Subsequently, we explore the underlying reasons for the high computational overhead of DL through reviewing the fundamental concepts including capacity, generalization, and backpropagation of a neural network. Guided by these concepts, we investigate on principles of representative research works, as well as three types of solutions: algorithmic design, computational optimization, and hardware revolution. In pursuant to these solutions, we identify challenges to be addressed.

GPU-accelerated discrete element modeling of geological faults

In this paper, we present an original algorithm for the numerical simulation of tectonic movements and the related formation of geological faults. The approach is based on the use of Discrete Element Method, where the geological media is represented as an agglomeration of discrete particles which interact as elastic, visco-elastic, or elasto-plastic bodies. This approach naturally allows accounting for large deformations and discontinuities in the geological media; thus, allows simulation of faults formation. Implementation of the algorithm is based on the use of Graphical Processor Units. The Discrete Element Method requires a high number of floating point operations and logical operations per a single particle (degree of freedom) per time step; whereas the number of memory access operations is relatively low. Thus, the use of GPUs decreases the computational time substantially.

Systolic hand gesture recognition/detection system based on FPGA with multi-port BRAMs

A new systolic approach of real-time vision-based hand gesture recognition is proposed. Hand gesture recognition is classified into two type, static and dynamic. The proposed system can be employed in both types. The system can detect and classify either a static hand shape in one image or a static hand shape at the start, within, and the end of gesture in a dynamic sequence in video stream. An efficient template matching (TM) based similarity measure is used to recognize gesture within the inter frame sequence of video frames. The template can be used to store the static class of different hand gesture shapes, while fast scanning approach of window over an image frame dynamically detects or recognizes the hand gesture. A suitable hardware systolic model of TM is designed. The systolic architecture is expanded to be used with multiple windows (k-templates) simultaneously. The required templates are equal to the number (k) of gestures that is to be recognized. They are stored in FPGA RAM blocks (BRAMs). To increase the bandwidth of templates BRAMs, the BRAM is multi-ported. A considerable reduction of memory access operations is achieved for the parallel systolic model in comparison to sequential/parallel software models. Consequently, the hand gesture is detected, tracked and recognized within the inter frame space of video. An acceptable recognition accuracy of up to 95% is achieved under variant observation conditions of three hand gesture classes dataset. A high frame rate video can be processed in real time.

A Face Alignment Accelerator Based on Optimized Coarse-to-Fine Shape Searching

The coarse-to-fine shape searching (CFSS) framework is a recently developed algorithm that achieves relatively high accuracy in face alignment by alleviating the poor initialization problem facing traditional cascaded regression approaches. However, its high computational complexity and memory access demands make it difficult for CFSS to satisfy the requirements of real-time processing. To address this issue, a fast shape searching face alignment (F-SSFA) accelerator is presented based on the optimization of the CFSS algorithm and an efficient hardware implementation. First, the learning-based low-dimensional speeded-up robust features method, based on the correlations between the SURF features and the regression targets, is introduced to distill the feature set down to the only most distinct features to reduce the computing load. Second, the partial keypoints Euclidean distance and shape affine transformation are introduced to replace feature extraction and support vector machine classification, thereby accelerating the shape searching process. Compared with CFSS, F-SSFA achieves a $5.8\times $ speedup while achieving similar accuracy. Moreover, a VLSI architecture is proposed to realize the fixed-point F-SSFA algorithm. Multiple descriptors located in adjacent regions are simultaneously generated in a single access to the corresponding image data. Therefore, repeated memory access operations are avoided. The optimal parameter configuration for hardware implementation is also exploited based on a tradeoff between accuracy and hardware performance. Simulated with TSMC 65-nm 1P8M technology within a 3.6 mm2 area, a post-layout simulation shows that 700 fps can be achieved while consuming 300 mW at 200 MHz.

Cross-point Resistive Memory

Emerging computational resistive memory is promising to overcome the challenges of scalability and energy efficiency that DRAM faces and also break through the memory wall bottleneck. However, cell-level and array-level nonideal properties of resistive memory significantly degrade the reliability, performance, accuracy, and energy efficiency during memory access and analog computation. Cell-level nonidealities include nonlinearity, asymmetry, and variability. Array-level nonidealities include interconnect resistance, parasitic capacitance, and sneak current. This review summarizes practical solutions that can mitigate the impact of nonideal device and circuit properties of resistive memory. First, we introduce several typical resistive memory devices with focus on their switching modes and characteristics. Second, we review resistive memory cells and memory array structures, including 1T1R, 1R, 1S1R, 1TnR, and CMOL. We also overview three-dimensional (3D) cross-point arrays and their structural properties. Third, we analyze the impact of nonideal device and circuit properties during memory access and analog arithmetic operations with focus on dot-product and matrix-vector multiplication. Fourth, we discuss the methods that can mitigate these nonideal properties by static parameter and dynamic runtime co-optimization from the viewpoint of device and circuit interaction. Here, dynamic runtime operation schemes include line connection, voltage bias, logical-to-physical mapping, read reference setting, and switching mode reconfiguration. Then, we highlight challenges on multilevel cell cross-point arrays and 3D cross-point arrays during these operations. Finally, we investigate design considerations of memory array peripheral circuits. We also portray an unified reconfigurable computational memory architecture.

Efficient implementation of MPI-3 RMA over openFabrics interfaces

The Message Passing Interface (MPI) standard supports Remote Memory Access (RMA) operations, where a process can read or write memory of another process without requiring the target process to be involved in the communication. This enables new more efficient programming models.This paper describes the RMA design and implementation in MPICH-OFI, an MPICH-based open source implementation of the MPI standard that uses the OpenFabrics Interfaces* (OFI*) to communicate with the underlying network fabric. MPICH-OFI is based on a new communication layer called CH4, which was designed to achieve high performance by minimizing the runtime software overhead and by having an internal API that is well aligned with MPI functions. MPICH-OFI uses the OpenFabrics Interfaces (OFI), a lightweight communication framework to support modern high-speed interconnects. Thanks to CH4 and OFI, MPICH-OFI achieves low latency and high bandwidth for RMA operations. Our experimental results using microbenchmarks show that MPICH-OFI achieves more than 3x better put/get latency and bandwidth than MPICH CH3, 10% better latency than Open MPI and MVAPICH2, and more than 1.7x bandwidth than MVAPICH2 for small messages ( ≤ 4KB), on Intel® Omni-Path Architecture.

An STT-MRAM Based in Memory Architecture for Low Power Integral Computing

The integral histogram image plays an important role in accelerating the feature computation in vision algorithms. However, the computational process of the integral histogram, called integral computation, has high computational complexity and numerous memory access operations, which limit its wide application. This brief proposes an in-memory computational architecture based on Spin Transfer Torque Magnetic Random Access Memory (STT-MRAM) to solve these problems. The architecture can work in two different modes depending on the requirements: the integral computation mode and the memory mode. The architecture can figure out the integral histogram when in the integral computation mode, and just store the data directly when in the memory mode. Utilizing the non-volatile, high density and low power characteristics of STT-MRAM, we integrate the computational units into the memory array to achieve parallel computation. Reduced number of data transmission between storage units and computation units contributes to cut down the latency and energy consumption. The evaluation results show that, comparing with the state-of-the-art work, our architecture provides <inline-formula><tex-math notation="LaTeX">$1.1\times \sim 9\times$</tex-math></inline-formula> performance improvements and reduces 87.4 <inline-formula><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 97.3 percent energy consumption for <inline-formula><tex-math notation="LaTeX">$64\times 64\sim 512\times 512$</tex-math></inline-formula> size images, just with a 8 percent area overhead.

Hardware-Enhanced Protection for the Runtime Data Security in Embedded Systems

At present, the embedded systems are facing various kinds of attacks, especially for the data stored in the external memories. This paper presents a hardware-enhanced protection method to protect the data integrity and confidentiality at runtime, preventing the data from spoofing attack, splicing attack, replay attack, and some malicious analysis. For the integrity protection, the signature is calculated by the hardware implemented Lhash engine before the data sending off the chip, and the signature of the data block is recalculated and compared with the decrypted one at the load time. For the confidentiality protection, an AES encryption engine is used to generate the key stream, the plain data and the cipher data can translate through a simple XOR operation. The hardware cryptographic engines are optimized to work simultaneously with the memory access operation, which reduces the hardware overhead and the performance overhead. We implement the proposed architecture within OR1200 processor on Xilinx Virtex 5 FPGA platform. The experiment results show that the proposed hardware-enhanced protection method can preserve the integrity and confidentiality of the runtime data in the embedded systems with low power consumption and a marginal area footprint. The performance overhead is less than 2.27% according to the selected benchmarks.

HPC Process and Optimal Network Device Affinitization

High Performance Computing (HPC) applications have demanding need for hardware resources such as processor, memory, and storage. Applications in the area of Artificial Intelligence and Machine Learning are taking center stage in HPC, which is driving demand for increasing compute resources per node which in turn is pushing bandwidth requirement between the compute nodes. New system design paradigms exist where deploying a system with more than one high performance IO device per node provides benefits. The number of I/O devices connected to the HPC node can be increased with PCIe switches and hence some of the HPC nodes are designed to include PCIe switches to provide a large number of PCIe slots. With multiple IO devices per node, application programmers are forced to consider HPC process affinity to not only compute resources but extend this to include IO devices. Mapping of process to processor cores and the closest IO device(s) increases complexity due to three way mapping and varying HPC node architectures. While operating systems perform reasonable mapping of process to processor core(s), they lack the application developer's knowledge of process workflow and optimal IO resource allocation when more than one IO device is attached to the compute node. This paper is an extended version of our work published in [1] . Our previous work provided solution for IO device affinity choices by abstracting the device selection algorithm from HPC applications. In this paper, we extend the affinity solution to enable OpenFabric Interfaces (OFI) which is a generic HPC API designed as part of the OpenFabrics Alliance that enables wider HPC programming models and applications supported by various HPC fabric vendors. We present a solution for IO device affinity choices by abstracting the device selection algorithm from HPC applications. MPI continues to be the dominant programming model for HPC and hence we provide evaluation with MPI based micro benchmarks. Our solution is then extended to OpenFabric Interfaces which supports other HPC programming models such as SHMEM, GASNet, and UPC. We propose a solution to solve NUMA issues at the lower level of the software stack that forms the runtime for MPI and other programming models independent of HPC applications. Our experiments are conducted on a two node system where each node consists of two socket Intel Xeon servers, attached with up to four Intel Omni-Path fabric devices connected over PCIe. The performance benefits seen by applications by affinitizing processes with best possible network device is evident from the results where we notice up to 40 percent improvement in uni-directional bandwidth, 48 percent bi-directional bandwidth, 32 percent improvement in latency measurements, and up to 40 percent improvement in message rate with OSU benchmark suite. We also extend our evaluation to include OFI operations and an MPI benchmark used for Genome assembly. With OFI Remote Memory Access (RMA) operations we see a bandwidth improvement of 32 percent for fi_read and 22 percent with fi_write operations, and also latency improvement of 15 percent for fi_read and 14 percent for fi_write. K-mer MMatching Interface HASH benchmark shows an improvement of up to 25 percent while using local network device versus using a network device connected to remote Xeon socket.