Memory Bandwidth Requirements Research Articles

A Systolic Accelerator for Neuromorphic Visual Recognition

Advances in neuroscience have encouraged researchers to focus on developing computational models that behave like the human brain. HMAX is one of the potential biologically inspired models that mimic the primate visual cortex’s functions and structures. HMAX has shown its effectiveness and versatility in multi-class object recognition with a simple computational structure. It is still a challenge to implement the HMAX model in embedded systems due to the heaviest computational S2 phase of HMAX. Previous implementations such as CoRe16 have used a reconfigurable two-dimensional processing element (PE) array to speed up the S2 layer for HMAX. However, the adder tree mechanism in CoRe16 used to produce output pixels by accumulating partial sums in different PEs increases the runtime for HMAX. To speed up the execution process of the S2 layer in HMAX, in this paper, we propose SAFA (systolic accelerator for HMAX), a systolic-array based architecture to compute and accelerate the S2 stage of HMAX. Using the output stationary (OS) dataflow, each PE in SAFA not only calculates the output pixel independently without additional accumulation of partial sums in multiple PEs, but also reduces the multiplexers applied in reconfigurable accelerators. Besides, data forwarding for the same input or weight data in OS reduces the memory bandwidth requirements. The simulation results show that the runtime of the heaviest computational S2 stage in HMAX model is decreased by 5.7%, and the bandwidth required for memory is reduced by 3.53 × on average by different kernel sizes (except for kernel = 12) compared with CoRe16. SAFA also obtains lower power and area costs than other reconfigurable accelerators from synthesis on ASIC.

Efficient Adaptive Deferred Shading with Hardware Scatter Tiles

Adaptive shading is an effective mechanism for reducing the number of shaded pixels to a subset of the image resolution with minimal impact on final rendering quality. We present a new scheduling method based on on-chip tiles that, along with relatively minor modifications to the GPU architecture, provides efficient hardware support. As compared to software implementations on current hardware using compute shaders, our approach dramatically reduces memory bandwidth requirements, thereby significantly improving performance and energy use. We also introduce the concept of a fragment pre-shader for programmatically controlling when a fragment shader is invoked, and describe advanced techniques for utilizing our approach to further reduce the number of shaded pixels via temporal filtering, or to adjust rendering quality to maintain stable framerates.

Efficient Hardware Architectures for 1D- and MD-LSTM Networks

Recurrent Neural Networks, in particular One-dimensional and Multidimensional Long Short-Term Memory (1D-LSTM and MD-LSTM) have achieved state-of-the-art classification accuracy in many applications such as machine translation, image caption generation, handwritten text recognition, medical imaging and many more. However, high classification accuracy comes at high compute, storage, and memory bandwidth requirements, which make their deployment challenging, especially for energy-constrained platforms such as portable devices. In comparison to CNNs, not so many investigations exist on efficient hardware implementations for 1D-LSTM especially under energy constraints, and there is no research publication on hardware architecture for MD-LSTM. In this article, we present two novel architectures for LSTM inference: a hardware architecture for MD-LSTM, and a DRAM-based Processing-in-Memory (DRAM-PIM) hardware architecture for 1D-LSTM. We present for the first time a hardware architecture for MD-LSTM, and show a trade-off analysis for accuracy and hardware cost for various precisions. We implement the new architecture as an FPGA-based accelerator that outperforms NVIDIA K80 GPU implementation in terms of runtime by up to 84× and energy efficiency by up to 1238× for a challenging dataset for historical document image binarization from DIBCO 2017 contest, and a well known MNIST dataset for handwritten digits recognition. Our accelerator demonstrates highest accuracy and comparable throughput in comparison to state-of-the-art FPGA-based implementations of multilayer perceptron for MNIST dataset. Furthermore, we present a new DRAM-PIM architecture for 1D-LSTM targeting energy efficient compute platforms such as portable devices. The DRAM-PIM architecture integrates the computation units in a close proximity to the DRAM cells in order to maximize the data parallelism and energy efficiency. The proposed DRAM-PIM design is 16.19 × more energy efficient as compared to FPGA implementation. The total chip area overhead of this design is 18 % compared to a commodity 8 Gb DRAM chip. Our experiments show that the DRAM-PIM implementation delivers a throughput of 1309.16 GOp/s for an optical character recognition application.

Write Back Energy Optimization for STT-MRAM-based Last-level Cache with Data Pattern Characterization

Traditional memory technologies face severe challenges in meeting the ever-increasing power and memory bandwidth requirements for high-performance computing and big-data analyses. Several emerging memory technologies are promising as the replacements of SRAM or DRAM. Among them, STT-MRAM can be used to replace SRAM as the last-level cache (LLC). However, it suffers from high write energy and latency. In this article, we investigate data patterns written from SRAM-based upper-level cache to STT-MRAM-based LLC to explore the write energy reduction potential. Depending on the data layout within a cache line, redundant bits can be identified and eliminated from write back operations to save STT-MRAM write energy. We also propose a dynamic profiling method to accommodate different application characteristics. The extensive simulation results show that write energy can be saved by 37.05% ∼ 38.89% for static profiling and 19.76% ∼ 34.29% for dynamic profiling.

A fast and scalable architecture to run convolutional neural networks in low density FPGAs

Deep learning and, in particular, convolutional neural networks (CNN) achieve very good results on several computer vision applications like security and surveillance, where image and video analysis are required. These networks are quite demanding in terms of computation and memory and therefore are usually implemented in high-performance computing platforms or devices. Running CNNs in embedded platforms or devices with low computational and memory resources requires a careful optimization of system architectures and algorithms to obtain very efficient designs. In this context, Field Programmable Gate Arrays (FPGA) can achieve this efficiency since the programmable hardware fabric can be tailored for each specific network. In this paper, a very efficient configurable architecture for CNN inference targeting any density FPGAs is described. The architecture considers fixed-point arithmetic and image batch to reduce computational, memory and memory bandwidth requirements without compromising network accuracy. The developed architecture supports the execution of large CNNs in any FPGA devices including those with small on-chip memory size and logic resources. With the proposed architecture, it is possible to infer an image in AlexNet in 4.3 ms in a ZYNQ7020 and 1.2 ms in a ZYNQ7045.

CIMAT: A Compute-In-Memory Architecture for On-chip Training Based on Transpose SRAM Arrays

Rapid development in deep neural networks (DNNs) is enabling many intelligent applications. However, on-chip training of DNNs is challenging due to the extensive computation and memory bandwidth requirements. To solve the bottleneck of the memory wall problem, compute-in-memory (CIM) approach exploits the analog computation along the bit line of the memory array thus significantly speeds up the vector-matrix multiplications. So far, most of the CIM-based architectures target at implementing inference engine for offline training only. In this article, we propose CIMAT, a CIM Architecture for Training. At the bitcell level, we design two versions of 7T and 8T transpose SRAM to implement bi-directional vector-to-matrix multiplication that is needed for feedforward (FF) and backprogpagation (BP). Moreover, we design the periphery circuitry, mapping strategy and the data flow for the BP process and weight update to support the on-chip training based on CIM. To further improve training performance, we explore the pipeline optimization of proposed architecture. We utilize the mature and advanced CMOS technology at 7 nm to design the CIMAT architecture with 7T/8T transpose SRAM array that supports bi-directional parallel read. We explore the 8-bit training performance of ImageNet on ResNet-18, showing that 7T-based design can achieve 3.38× higher energy efficiency (∼6.02 TOPS/W), 4.34× frame rate (∼4,020 fps) and only 50 percent chip size compared to the baseline architecture with conventional 6T SRAM array that supports row-by-row read only. The even better performance is obtained with 8T-based architecture, which can reach ∼10.79 TOPS/W and ∼48,335 fps with 74-percent chip area compared to the baseline.

Res-DNN: A Residue Number System-Based DNN Accelerator Unit

In this article, a technique, based on using Residue Number System (RNS) is suggested to improve the energy efficiency of Deep Neural Networks (DNNs). In the DNN architecture, which is fully RNS-based, only weights and the primary inputs in the main memory are in the binary number system (BNS). The architecture, which is called Res-DNN, offers a high energy saving while requiring higher bit count for data to handle the overflow compared to that of a BNS one. Scaling techniques in the processing elements are employed in the RNS-based computations to make the computation bit widths the same as the BNS bit width. In this architecture, the MAX pooling and ReLU activation function are implemented in the RNS format. To lower the memory usage and required memory bandwidth, we suggest a Huffman-based coding. Additionally, for accessing the weights stored in the main memory, to obtain further energy reduction, we propose a structural modification to the memory hierarchy where a lower level register file is added to the data path of these accesses. The effectiveness of the proposed architecture is evaluated under seven state-of-the-art DNNs with the datasets of ImageNet and CIFAR-10. The obtained results show that Res-DNN leads to $2.5\times $ lower energy for computations and an average of 30% overall energy reduction compared to those of the binary counterpart.

High-Performance FPGA-Based CNN Accelerator With Block-Floating-Point Arithmetic

Convolutional neural networks (CNNs) are widely used and have achieved great success in computer vision and speech processing applications. However, deploying the large-scale CNN model in the embedded system is subject to the constraints of computation and memory. An optimized block-floating-point (BFP) arithmetic is adopted in our accelerator for efficient inference of deep neural networks in this paper. The feature maps and model parameters are represented in 16-bit and 8-bit formats, respectively, in the off-chip memory, which can reduce memory and off-chip bandwidth requirements by 50% and 75% compared to the 32-bit FP counterpart. The proposed 8-bit BFP arithmetic with optimized rounding and shifting-operation-based quantization schemes improves the energy and hardware efficiency by three times. One CNN model can be deployed in our accelerator without retraining at the cost of an accuracy loss of not more than 0.12%. The proposed reconfigurable accelerator with three parallelism dimensions, ping-pong off-chip DDR3 memory access, and an optimized on-chip buffer group is implemented on the Xilinx VC709 evaluation board. Our accelerator achieves a performance of 760.83 GOP/s and 82.88 GOP/s/W under a 200-MHz working frequency, significantly outperforming previous accelerators.

Area–Delay–Energy Efficient VLSI Architecture for Scalable In-Place Computation of FFT on Real Data

Efficient computation of real-valued fast Fourier transform (RFFT) has received significant attention in recent years due to its several applications in conventional digital signal processing and other emerging areas. In-place RFFT architectures are gaining popularity due to their lower hardware complexity compared with pipeline architectures. But the scaling of in-place RFFT architecture for higher lengths and higher throughput is a challenging issue due to increasing memory access conflict and higher memory bandwidth requirement. In this paper, a design approach is presented to develop an area-delay and energy-efficient architecture for in-place RFFT. Generally, an in-place fast Fourier transform (FFT) structure consists of a butterfly block which performs a set of butterfly operations in every clock cycle. From complexity analysis we find that in-place FFT structures with larger butterfly blocks are more efficient in terms of area-time complexity and energy consumption. The resolution of memory access conflict is however more challenging for higher butterfly block sizes. Therefore, we have analyzed the data-flow and memory footprint of in-place RFFT architectures for different throughput requirements, and based on that, we have proposed here a strategy to partition the storage unit into several banks of smaller sizes (without increasing the overall memory size) to resolve the memory access conflicts by concurrent data-swapping between the banks. Synthesis result shows that the proposed structure with butterfly block of size 4 and 8 involves (~44% and ~57%) less area-delay product and (~54% and ~57%) less energy per sample than those of existing similar structure on average for different FFT lengths, respectively.

A 99.4 fps Optical Flow Estimation Processor with Image Tiling for Action Recognition in Mobile Devices

A 99.4 fps optical flow estimation (OFE) processor with image tiling is proposed for action recognition in mobile devices. The OFE is essential for the high action recognition accuracy. However, it is unsuitable for real-time constraint in a mobile computing environment because it requires a huge amount of external memory accesses (EMAs) and matrix computations. For mitigating the external memory bandwidth requirement, this paper proposes the tile-based hierarchical OFE. It divides input images into several tiles and enables intermediate data reusing with 326.4 KB on-chip memory and 175.8 MB/s external memory bandwidth. Moreover, a background decision unit with early termination is proposed to reduce computation workload. It gets rid of unnecessary matrix computation by terminates the computation early for zero optical flow region. As a result, the proposed features reduce external memory bandwidth by 99.3 % and increase throughput by 50.7 %, respectively. The proposed 12.8 mm2 OFE processor is implemented in 65 nm CMOS technology, and it achieves the real-time OFE with 99.4 framesper-second (fps) throughput for an image resolution of QVGA (320 × 240).

A Hardware-Efficient Block Matching Algorithm and Its Hardware Design for Variable Block Size Motion Estimation in Ultra-High-Definition Video Encoding

Variable block size motion estimation has contributed greatly to achieving an optimal interframe encoding, but involves high computational complexity and huge memory access, which is the most critical bottleneck in ultra-high-definition video encoding. This article presents a hardware-efficient block matching algorithm with an efficient hardware design that is able to reduce the computational complexity of motion estimation while providing a sustained and steady coding performance for high-quality video encoding. A three-level memory organization is proposed to reduce memory bandwidth requirement while supporting a predictive common search window. By applying multiple search strategies and early termination, the proposed design provides 1.8 to 3.7 times higher hardware efficiency than other works. Furthermore, on-chip memory has been reduced by 96.5% and off-chip bandwidth requirement has been reduced by 39.4% thanks to the proposed three-level memory organization. The corresponding power consumption is only 198mW at the highest working frequency of 500MHz. The proposed design is attractive for high-quality video encoding in real-time applications with low power consumption.

SynergyFlow

Neural networks (NNs) have achieved great success in a broad range of applications. As NN-based methods are often both computation and memory intensive, accelerator solutions have been proved to be highly promising in terms of both performance and energy efficiency. Although prior solutions can deliver high computational throughput for convolutional layers, they could incur severe performance degradation when accommodating the entire network model, because there exist very diverse computing and memory bandwidth requirements between convolutional layers and fully connected layers and, furthermore, among different NN models. To overcome this problem, we proposed an elastic accelerator architecture, called SynergyFlow, which intrinsically supports layer-level and model-level parallelism for large-scale deep neural networks. SynergyFlow boosts the resource utilization by exploiting the complementary effect of resource demanding in different layers and different NN models. SynergyFlow can dynamically reconfigure itself according to the workload characteristics, maintaining a high performance and high resource utilization among various models. As a case study, we implement SynergyFlow on a P395-AB FPGA board. Under 100MHz working frequency, our implementation improves the performance by 33.8% on average (up to 67.2% on AlexNet) compared to comparable provisioned previous architectures.

Three-level pipelined multi-resolution integer motion estimation engine with optimized reference data sharing search for AVS

Integer motion estimation (IME), which acts as a key component in video encoder, is to remove temporal redundancies by searching the best integer motion vectors for dynamic partition blocks in a macro-block (MB). Huge memory bandwidth requirements and unbearable computational resource demanding are two key bottlenecks in IME engine design, especially for large search window (SW) cases. In this paper, a three-level pipelined VLSI architecture design is proposed, where efficiently integrates the reference data sharing search (RDSS) into multi-resolution motion estimation algorithm (MMEA). First, a hardware-friendly MMEA algorithm is mapped into three-level pipelined architecture with neglected coding quality loss. Second, sub-sampled RDSS coupled with Level C + are adopted to reduce on-chip memory and bandwidth at the coarsest and middle level. Data sharing between IME and fractional motion estimation (FME) is achieved by loading only a local predictive SW at the finest level. Finally, the three levels are parallelized and pipelined to guarantee the gradual refinement of MMEA and the hardware utilization. Experimental results show that the proposed architecture can reach a good balance among complexity, on-chip memory, bandwidth, and the data flow regularity. Only 320 processing elements (PE) within 550 cycles are required for IME search, where the SW is set to 256 × 256. Our architecture can achieve 1080P@30 fps real-time processing at the working frequency of 134.6 MHz, with 135 K gates and 8.93 KB on-chip memory.

Runtime Programmable and Memory Bandwidth Optimized FPGA-Based Coprocessor for Deep Convolutional Neural Network.

The deep convolutional neural network (DCNN) is a class of machine learning algorithms based on feed-forward artificial neural network and is widely used for image processing applications. Implementation of DCNN in real-world problems needs high computational power and high memory bandwidth, in a power-constrained environment. A general purpose CPU cannot exploit different parallelisms offered by these algorithms and hence is slow and energy inefficient for practical use. We propose a field-programmable gate array (FPGA)-based runtime programmable coprocessor to accelerate feed-forward computation of DCNNs. The coprocessor can be programmed for a new network architecture at runtime without resynthesizing the FPGA hardware. Hence, it acts as a plug-and-use peripheral for the host computer. Caching is implemented for input features and filter weights using on-chip memory to reduce the external memory bandwidth requirement. Data are prefetched at several stages to avoid stalling of computational units and different optimization techniques are used to efficiently reuse the fetched data. Dataflow is dynamically adjusted in runtime for each DCNN layer to achieve consistent computational throughput across a wide range of input feature sizes and filter sizes. The coprocessor is prototyped using Xilinx Virtex-7 XC7VX485T FPGA-based VC707 board and operates at 150 MHz. Experimental results show that our implementation is energy efficient than highly optimized CPU implementation and achieves consistent computational throughput of more than 140 G operations/s for a wide range of input feature sizes and filter sizes. Off-chip memory transactions decrease by due to the use of the on-chip cache.

Architecture Design of a Memory Subsystem for Massive MIMO Baseband Processing

This brief presents an on-chip memory subsystem for massive multiple-input-multiple-output (MIMO) baseband processing at the base station. In massive MIMO systems, the required memory bandwidth and capacity are orders of magnitude higher than those used in conventional wireless systems, due to the large number of serving antennas. These are further combined with design targets on low access latency and flexibility in data organization and access modes. This brief applies and improves the concept of parallel memories to achieve the challenging design target with low hardware overhead. As a case study, a memory subsystem for 128-antenna and 16-user massive MIMO systems is evaluated using ST 28-nm technology. According to postlayout simulation results, the proposed memory subsystem provides 512-Gb/s throughput and offers 1-Mb capacity with a cost of 0.30 mm2.

UniCNN: A Pipelined Accelerator Towards Uniformed Computing for CNNs

Convolutional neural networks (CNNs) have been widely applied for image recognition, face detection, and video analysis because of their ability to achieve accuracy close to or even better than human level perception. However, different features of convolution layers and fully connected layers have brought many challenges to the implementation of CNN on FPGA platforms, because different accelerator units must be designed to process the whole networks. In order to overcome this problem, this work proposes a pipelined accelerator towards uniformed computing for convolutional neural networks. For the convolution layer, the accelerator first repositions the input features into matrix on-the-fly when they are stored to FPGA on-chip buffers, thus the computation of convolution layer can be completed through matrix multiplication. For the fully connected layer, the batch-based method is used to reduce the required memory bandwidth, which also can be completed through matrix multiplication. Then a pipelined computation method for matrix multiplication is proposed to increase the throughput and also reduce the buffer requirement. The experiment results show that the proposed accelerator surpasses CPUs and GPUs platform in terms of energy efficiency. The proposed accelerator can achieve the throughput of 49.31 GFLOPS, which is done using only 198 DSP modules. Compared to the state-of-the-art implementatuion, our accelerator has better hardware utilization efficiency.

Z2 traversal order: An interleaving approach for VR stereo rendering on tile-based GPUs

With increasing demands of virtual reality (VR) applications, efficient VR rendering techniques are becoming essential. Because VR stereo rendering has increased computational costs to separately render views for the left and right eyes, to reduce the rendering cost in VR applications, we present a novel traversal order for tile-based mobile GPU architectures: Z2 traversal order. In tile-based mobile GPU architectures, a tile traversal order that maximizes spatial locality can increase GPU cache efficiency. For VR applications, our approach improves upon the traditional Z order curve. We render corresponding screen tiles in left and right views in turn, or simultaneously, and as a result, we can exploit spatial adjacency of the two tiles. To evaluate our approach, we conducted a trace-driven hardware simulation using Mesa and a hardware simulator. Our experimental results show that Z2 traversal order can reduce external memory bandwidth requirements and increase rendering performance.

Improving performance by network-aware virtual machine clustering and consolidation

Modern data center consists of thousands of servers, racks and switches. Complicated structure means it requires well-designed algorithms to utilize resources of data centers efficiently. Current virtual machine scheduling algorithms mainly focus on the initial allocation of virtual machines based on the CPU, memory and network bandwidth requirements. However, when tasks finished or lease expired, related virtual machines would be deleted from the system which would generate resource fragments. Such fragments lead to unbalanced resource utilization and decline of communication performance. This paper investigates the network influence on typical applications in data centers and proposed a self-adaptive network-aware virtual machine clustering and consolidation algorithm to maintain an optimal system-wide status. Our consolidation algorithm periodically checks whether consolidation is necessary and then clusters and consolidates virtual machines to lower communication cost with an online heuristic. We used two benchmarks in a real environment to examine network influence on different tasks. To evaluate the advantages of the proposed algorithm, we also built a cloud computing testbed. Real workload trace-driven simulations and testbed-based experiments showed that, our algorithm greatly shortened the average finish time of map-reduce tasks and reduced time delay of web applications. Simulation results showed that our algorithm considerably reduced the amount of high-delay jobs, lowered the average traffic passed through aggregate switches and improved the communication ability among virtual machines.

Surface Couplings for Subdomain-Wise Isoviscous Gradient Based Stokes Finite Element Discretizations

The Stokes system with constant viscosity can be cast into different formulations by exploiting the incompressibility constraint. For instance, the rate of strain tensor in the weak formulation can be replaced by the velocity-gradient yielding a decoupling of the velocity components in the different coordinate directions. Consequently, the discretization of this partly decoupled formulation leads to fewer nonzero entries in the stiffness matrix. This is of particular interest in large scale simulations where a reduced memory bandwidth requirement can help to significantly accelerate the computations. In the case of a piecewise constant viscosity, as it typically arises in multi-phase flows, or when the boundary conditions involve traction, the situation is more complex, and one has to treat the cross derivatives in the original Stokes system with care. A naive application of the standard vectorial Laplacian results in a physically incorrect solution, while formulations based on the rate of strain tensor increase the computational effort globally. Here, we propose a new approach that is consistent with the stress-divergence formulation and preserves the decoupling advantages of the velocity-gradient-divergence formulation in isoviscous subdomains. The modification is equivalent to locally changing the discretization stencils at interfaces or boundaries. Hence, the more expensive discretization is of lower complexity, making the additional computational cost in large scale simulations negligible. We establish consistency and convergence properties and show that in a massively parallel setup, the multigrid solution of the resulting discrete systems is faster than for the classical stress-divergence formulation.

Alleviate Chip Pin Constraint for Multicore Processor by On/Off-Chip Power Delivery System Codesign

The number of chip pins is limited due to the cost and reliability issues of sophisticated packages, and it is predicted that the chip pin count will be overstretched to satisfy the requirements of both power delivery and memory access. The gap between the achievable pin count and the demand will increase as the technology scales, due to the increasing computation resources and supply current. Pin reduction techniques are thus required for continued computing performance growth. In this article, we propose a chip pin constraint alleviation strategy, through on/off-chip power delivery system co-design, to effectively reduce the demand for power pins. An analytical model of a power delivery system, consisting of on/off-chip regulators and a power delivery network, is proposed to evaluate the influence of regulator design and package conduction loss. By combining this model with a multi-core processor model of performance and memory bandwidth requirements, we characterize the entire multi-core processor system to investigate the relationship between the chip pin constraint and performance in multi-core processor scaling and the effectiveness of our strategy. Experiments show that with the conventional power delivery system design, the chip pin constraint severely limits the performance growth as the technology scales. Using the on/off-chip power delivery system co-design, our strategy achieves a significant pin count reduction, for example, 31.3% at the 8nm technology node, compared to the conventional design with the same chip performance, while, provided with the same chip pin count, it is able to improve, by 35.0%, the chip performance at 8nm compared to the conventional design. For real applications of different parallelism, our strategy outperforms its counterpart, with a 23.7% performance improvement on average at the 8nm technology node.