Convolution Engine Research Articles

An Efficient FPGA-based Depthwise Separable Convolutional Neural Network Accelerator with Hardware Pruning

Convolutional neural networks (CNNs) have been widely deployed in computer vision tasks. However, the computation and resource intensive characteristics of CNN bring obstacles to its application on embedded systems. This article proposes an efficient inference accelerator on Field Programmable Gate Array (FPGA) for CNNs with depthwise separable convolutions. To improve the accelerator efficiency, we make four contributions: (1) an efficient convolution engine with multiple strategies for exploiting parallelism and a configurable adder tree are designed to support three types of convolution operations; (2) a dedicated architecture combined with input buffers is designed for the bottleneck network structure to reduce data transmission time; (3) a hardware padding scheme to eliminate invalid padding operations is proposed; and (4) a hardware-assisted pruning method is developed to support online tradeoff between model accuracy and power consumption. Experimental results show that for MobileNetV2 the accelerator achieves 10× and 6× energy efficiency improvement over the CPU and GPU implementation, and 302.3 frames per second and 181.8 GOPS performance that is the best among several existing single-engine accelerators on FPGAs. The proposed hardware-assisted pruning method can effectively reduce 59.7% power consumption at the accuracy loss within 5%.

Accelerating Image Processing Using Reduced Precision Calculation Convolution Engines

In this paper a method of accelerating image processing using convolution engines with reduced precision calculation is presented. The convolution engines are designed to be used with the Pulpissimo platform with RISC-V System-on-Chip. The aim is to move the calculation to the edge. The proposed linear convolution engines operate on 8-bit data set and the logarithmic convolution engine operates on 4-bit reduced precision data. The data reduction is done by using a logarithmic number space. Diminishing the size of the data to be processed reduces the amount of required memory, requirement for memory bandwidth, required computation, and required hardware area while simultaneously increasing the performance. This performance could benefit modern AI and image processing applications, especially in mobile and other battery-operated devices. The results show that the computation in the linear convolution engine is 91 times faster and computation in the logarithmic convolution engine is 122 times faster than in the RISC-V core with plain RISC-V instructions.

DycSe: A Low-Power, Dynamic Reconfiguration Column Streaming-Based Convolution Engine for Resource-Aware Edge AI Accelerators

Edge AI accelerators are utilized to accelerate the computation in edge AI devices such as image recognition sensors on robotics, door lockers, drones, and remote sensing satellites. Instead of using a general-purpose processor (GPP) or graphic processing unit (GPU), an edge AI accelerator brings a customized design to meet the requirements of the edge environment. The requirements include real-time processing, low-power consumption, and resource-awareness, including resources on field programmable gate array (FPGA) or limited application-specific integrated circuit (ASIC) area. The system’s reliability (e.g., permanent fault tolerance) is essential if the devices target radiation fields such as space and nuclear power stations. This paper proposes a dynamic reconfigurable column streaming-based convolution engine (DycSe) with programmable adder modules for low-power and resource-aware edge AI accelerators to meet the requirements. The proposed DycSe design does not target the FPGA platform only. Instead, it is an intellectual property (IP) core design. The FPGA platform used in this paper is for prototyping the design evaluation. This paper uses the Vivado synthesis tool to evaluate the power consumption and resource usage of DycSe. Since the synthesis tool is limited to giving the final complete system result in the designing stage, we compare DycSe to a commercial edge AI accelerator for cross-reference with other state-of-the-art works. The commercial architecture shares the competitive performance within the low-power ultra-small (LPUS) edge AI scopes. The result shows that DycSe contains 3.56% less power consumption and slight resources (1%) overhead with reconfigurable flexibility.

Streaming Dilated Convolution Engine

Convolution is one of the most critical operations in various application domains and its computation should combine high performance with energy efficiency. This requirement is critical both for standard convolution and for its other spatial variants, such as dilated, strided, or transposed convolutions. In this work, we focus on the design of a streaming convolution engine, called LazyDCstream, that is tuned for dilated convolution. LazyDCstream utilizes a sliding-window architecture for input data reuse and leverages the already-known decomposition of dilated convolution to: (a) maximize window buffer sharing and (b) enable “lazy” data movement that keeps data transfers per clock cycle as few as possible, and, most importantly, independent of the dilation rate. These two architectural features reduce the power consumption relative to efficient streaming convolution engines without introducing any throughput or area penalty.

XMA2: A crossbar-aware multi-task adaption framework via 2-tier masks

Recently, ReRAM crossbar-based deep neural network (DNN) accelerator has been widely investigated. However, most prior works focus on single-task inference due to the high energy consumption of weight reprogramming and ReRAM cells’ low endurance issue. Adapting the ReRAM crossbar-based DNN accelerator for multiple tasks has not been fully explored. In this study, we propose XMA2, a novel crossbar-aware learning method with a 2-tier masking technique to efficiently adapt a DNN backbone model deployed in the ReRAM crossbar for new task learning. During the XMA2-based multi-task adaption (MTA), the tier-1 ReRAM crossbar-based processing-element- (PE-) wise mask is first learned to identify the most critical PEs to be reprogrammed for essential new features of the new task. Subsequently, the tier-2 crossbar column-wise mask is applied within the rest of the weight-frozen PEs to learn a hardware-friendly and column-wise scaling factor for new task learning without modifying the weight values. With such crossbar-aware design innovations, we could implement the required masking operation in an existing crossbar-based convolution engine with minimal hardware/memory overhead to adapt to a new task. The extensive experimental results show that compared with other state-of-the-art multiple-task adaption methods, XMA2 achieves the highest accuracy on all popular multi-task learning datasets.

A Flexible and Efficient FPGA Accelerator for Various Large-Scale and Lightweight CNNs

To enable efficient deployment of convolutional neural networks (CNNs) on embedded platforms for different computer vision applications, several convolution variants have been introduced, such as depthwise convolution (DWCV), transposed convolution (TPCV), and dilated convolution (DLCV). To address the utilization degradation issue occurred in a general convolution engine for these emerging operators, a highly flexible and reconfigurable hardware accelerator is proposed to efficiently support various CNN-based vision tasks. Firstly, to avoid workload imbalance of TPCV, a zero transfer and skipping (ZTS) method is proposed to reorganize the computation process. To eliminate the redundant zero calculations of TPCV and DLCV, a sparsity-alike processing (SAP) method is proposed based on weight-oriented dataflow. Secondly, the DWCV or pooling layers are configured to be directly executed after standard convolutions without external memory accesses. Furthermore, a programmable execution schedule is introduced to gain better flexibility. Finally, the proposed accelerator is evaluated on Intel Arria 10 SoC FPGA. Experimental results show state-of-the-art performance on both large-scale and lightweight CNNs for image segmentation or classification. Specifically, the accelerator can achieve a processing speed up to 339.9 FPS and computational efficiency up to 0.58 GOPS/DSP, which is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.3\times $ </tex-math></inline-formula> better than the prior art evaluated on the same network.

Phase‐Transition‐Induced VO2 Thin Film IR Photodetector and Threshold Switching Selector for Optical Neural Network Applications

AbstractAs the architecture of choice for future artificial‐intelligent systems, the ideas of in‐memory‐ and in‐sensor‐computing paradigms based on non‐von‐Neumann architecture possess broad application prospects such as neuromorphic and sensor‐memory‐processor fusion systems. At the same time, these promising applications put diversified and strict requirements on the device performances, such as fast response to external signals, robust data security, and 3D integration potential. In this work, Au@VO2 IR photodetectors and Ti/Au/VO2/Ti/Au threshold switching selectors are constructed, where the VO2 thin films are realized by magnetron sputtering and water‐vapor assisted post‐annealing. Fast IR response is achieved in Au@VO2 photodetectors through a surface plasmon resonance‐assisted metal‐insulator transition. Furthermore, electroforming‐free, tunable threshold voltage, steep switching slope, and selectivity of more than two orders of magnitude are observed in Ti/Au/VO2/Ti/Au threshold switching selector. Combining the functionalities of photodetection and selector, a VO2‐based optical convolution engine demonstrates accurate and secure image‐processing capability. These VO2‐based devices are demonstrated as promising candidates for novel non‐volatile memory, neuromorphic computing and sensor‐memory‐processor fusion applications.

Efficient field‐programmable gate array‐based reconfigurable accelerator for deep convolution neural network

Deep convolutional neural networks (DCNNs) have been widely applied in various modern artificial intelligence (AI) applications. DCNN's inference is a process with high calculation costs, which usually requires billions of multiply-accumulate operations. On mobile platforms such as embedded systems or robotics, an efficient implementation of DCNNs is significant. However, most previous field-programmable gate array-based works on accelerators for DCNNs just support one DCNN or just support convolution layers. In order to address this limitation, this work proposes a reconfigurable accelerator. The accelerator is flexible and can support multiple DCNNs and different layer types, such as convolution, pooling, activation function, and full connection layers. It is equipped with a five-level pipeline convolution engine whose main component is two processing element arrays. Furthermore, a design space exploration method is proposed to make full advantage of the proposed accelerator. This accelerator is implemented with the ZYNQ-7 ZC706 evaluation board and achieves a high performance of 53.29 Giga operations per second (GOPS) on AlexNet and 45.09 GOPS on YOLOv2-tiny at 100 MHz. Further performance of the accelerator is compared with the previous works, and it achieves multiple advantages: High performance, high configurability, and efficient resource utilisation.

FFConv

Image classification is known to be one of the most challenging problems in the domain of computer vision. Significant research is being done on developing systems and algorithms improving accuracy, performance, area, and power consumption for related problems. Convolutional Neural Networks (CNNs) have shown to give outstanding accuracies for problems such as image classification, object detection, and semantic segmentation. While CNNs are pioneering the development of high accuracy systems, their excessive computational complexity presents a barrier for a more permeated deployment. Although Graphical Processing Units (GPUs), due to their massively parallel architecture, have shown to give performance orders of magnitude better than general purpose processors, the former are limited by their high power consumption and generality. Consequently, Field Programmable Gate Arrays (FPGAs) are being explored to implement CNN architectures, as they also provide massively parallel logic resources but with a relatively lower power consumption than GPUs. In this article, we present FFConv, an efficient FPGA-based fast convolutional layer accelerator for CNNs. We design a pipelined, high-throughput convolution engine based on the Winograd minimal filtering (also called Fast Convolution) algorithms for computing the convolutional layers of three popular CNN architectures: VGG16, Alexnet, and Shufflenet. We implement our accelerator on a Virtex-7 FPGA platform where we exploit the computational parallelization to the maximum while exploring optimizations aimed at improving performance. The resultant design loses only 0.43%, 0.47%, and 0.61% Top-1 classification accuracy for VGG16, Alexnet, and Shufflenet-v1, respectively, while significantly improving throughput, resource, and power efficiency compared to previous state-of-the-art designs.

What is the correct ear-canal boundary condition for simulating HRTFs for open-ear listening?

Emerging open-ear binaural reproduction methods for virtual and augmented reality are an opportunity to rethink how we pose the simulation problem for head related transfer functions (HRTFs). Historically, researchers have experimentally measured HRTFs using a simplifying blocked meatus condition, then used those HRTFs to reproduce spatial audio using low-latency head tracking, an HRTF convolution engine, and carefully calibrated free-air-equivalent coupling over-the-ear headphones. This blocked meatus boundary condition was then carried over as researchers have moved toward numerical simulation of HRTFs, allowing for direct comparison of simulation and experiment. Now that the industry is trending toward numerically-simulated HRTFs and open-ear playback methods, a more realistic ear-canal input impedance needs to be further investigated. In this paper, we report changes in the sound pressure at the entrance to the ear-canal from open to blocked meatus conditions, using a Multiphysics Finite Element model. The blocked meatus boundary is replaced with a nominal frequency-dependent ear-canal input impedance and the pressure-division ratio of open to blocked condition is evaluated. These modeling results suggest that it may not be necessary to perform detailed geometry capture of the ear-canal and that a simplified input impedance is sufficient to simulate the open ear effect.

You Cannot Improve What You Do not Measure

Recently, deep learning (DL) has become best-in-class for numerous applications but at a high computational cost that necessitates high-performance energy-efficient acceleration. The reconfigurability of FPGAs is appealing due to the rapid change in DL models but also causes lower performance and area-efficiency compared to ASICs. In this article, we implement three state-of-the-art computing architectures (CAs) for convolutional neural network (CNN) inference on FPGAs and ASICs. By comparing the FPGA and ASIC implementations, we highlight the area and performance costs of programmability to pinpoint the inefficiencies in current FPGA architectures. We perform our experiments using three variations of these CAs for AlexNet, VGG-16 and ResNet-50 to allow extensive comparisons. We find that the performance gap varies significantly from 2.8× to 6.3×, while the area gap is consistent across CAs with an 8.7 average FPGA-to-ASIC area ratio. Among different blocks of the CAs, the convolution engine, constituting up to 60% of the total area, has a high area ratio ranging from 13 to 31. Motivated by our FPGA vs. ASIC comparisons, we suggest FPGA architectural changes such as increasing DSP block count, enhancing low-precision support in DSP blocks and rethinking the on-chip memories to reduce the programmability gap for DL applications.

NEURA ghe

Deep convolutional neural networks (CNNs) obtain outstanding results in tasks that require human-level understanding of data, like image or speech recognition. However, their computational load is significant, motivating the development of CNN-specialized accelerators. This work presents NEURA ghe , a flexible and efficient hardware/software solution for the acceleration of CNNs on Zynq SoCs. NEURA ghe leverages the synergistic usage of Zynq ARM cores and of a powerful and flexible Convolution-Specific Processor deployed on the reconfigurable logic. The Convolution-Specific Processor embeds both a convolution engine and a programmable soft core, releasing the ARM processors from most of the supervision duties and allowing the accelerator to be controlled by software at an ultra-fine granularity. This methodology opens the way for cooperative heterogeneous computing: While the accelerator takes care of the bulk of the CNN workload, the ARM cores can seamlessly execute hard-to-accelerate parts of the computational graph, taking advantage of the NEON vector engines to further speed up computation. Through the companion NeuDNN SW stack, NEURA ghe supports end-to-end CNN-based classification with a peak performance of 169GOps/s, and an energy efficiency of 17GOps/W. Thanks to our heterogeneous computing model, our platform improves upon the state-of-the-art, achieving a frame rate of 5.5 frames per second (fps) on the end-to-end execution of VGG-16 and 6.6fps on ResNet-18.

인공신경망 연산에 최적화 된 Switched Capacitor 구조의 저전력 8-bit 합성곱 연산기

본 논문에서는 기존의 디지털 방식이 아닌 아날로그 도메인에서 합성곱 연산을 수행하며 저해상도, 저전력, 고효율의 특성을 갖는 신경망 연산기에 대한 연구를 소개한다. 제안하는 연산기는 적분기(integrator) 구조의 곱셈 디지털-아날로그 변환기(multiplying DAC, MDAC)와 축차비교 아날로그-디지털 변환기(Successive-Approximation ADC, SAR ADC)로 구성되어 있다. 동작할 때 곱셈연산을 하는 동시에 덧셈연산이 이루어지며, opamp 출력단에 전하 형태로 적분하여 덧셈연산을 하므로 디지털 연산에 비해 메모리 접근 빈도가 낮은 장점을 갖고 있다. 65㎚ CMOS공정을 이용하여 MDAC과 ADC로 구성된 디지털-입력, 디지털-출력 연산기를 설계하였고, 트랜지스터-레벨 시뮬레이션 결과 33.3㎒의 속도로 30.11㎼의 전력을 소모하였고, 이는 2.21TOPS/W의 연산효율과 같아 기존의 디지털 방식의 합성곱 연산기보다 개선된 전력 효율을 나타내었다.

Virtual acoustics in multimedia production—Beyond enhancing the acoustics of concert halls

The paper describes a range of applications of virtual acoustics, the rendering of artificial acoustic spaces, which allow musicians to interact with ambient spaces created in real time. With a number of loudspeakers suitably distributed within a physical enclosure, such a projection system can be used to introduce a range of sound fields, which may effectively transform the acoustic environment to become a creative partner in multimedia production. A necessary component in this system is a low-latency, high-resolution multichannel convolution engine that converts a live audio signal into a structured ambient response, creating a scene in real time. Scenes can be changed to suit various goals of production and sonic narration. A number of techniques have been used to capture and modify impulse responses including temporal segmentation, shaping of magnitude envelope, noise reduction, spectral enrichment, time shifting and alignment, and parallel and sequential convolution. With these methods, artists may interact with novel acoustic responses as if they were musical instruments. Artists claim that many of the familiar as well as novel acoustic responses stimulate their creativity. Flexible variable virtual acoustics opens additional creative possibilities when employed as a component of a recording studio, either directly or when rendered over headphones.

Convolution engine

General-purpose processors, while tremendously versatile, pay a huge cost for their flexibility by wasting over 99% of the energy in programmability overheads. We observe that reducing this waste requires tuning data storage and compute structures and their connectivity to the data-flow and data-locality patterns in the algorithms. Hence, by backing off from full programmability and instead targeting key data-flow patterns used in a domain, we can create efficient engines that can be programmed and reused across a wide range of applications within that domain. We present the Convolution Engine (CE)---a programmable processor specialized for the convolution-like data-flow prevalent in computational photography, computer vision, and video processing. The CE achieves energy efficiency by capturing data-reuse patterns, eliminating data transfer overheads, and enabling a large number of operations per memory access. We demonstrate that the CE is within a factor of 2--3× of the energy and area efficiency of custom units optimized for a single kernel. The CE improves energy and area efficiency by 8--15× over data-parallel Single Instruction Multiple Data (SIMD) engines for most image processing applications.&lt;!-- END_PAGE_1 --&gt;

Convolution engine

This paper focuses on the trade-off between flexibility and efficiency in specialized computing. We observe that specialized units achieve most of their efficiency gains by tuning data storage and compute structures and their connectivity to the data-flow and data-locality patterns in the kernels. Hence, by identifying key data-flow patterns used in a domain, we can create efficient engines that can be programmed and reused across a wide range of applications. We present an example, the Convolution Engine (CE), specialized for the convolution-like data-flow that is common in computational photography, image processing, and video processing applications. CE achieves energy efficiency by capturing data reuse patterns, eliminating data transfer overheads, and enabling a large number of operations per memory access. We quantify the tradeoffs in efficiency and flexibility and demonstrate that CE is within a factor of 2-3x of the energy and area efficiency of custom units optimized for a single kernel. CE improves energy and area efficiency by 8-15x over a SIMD engine for most applications.

Active field control using sound field generation technique—Case study of a Live concert at a virtual Renaissance church

In the Renaissance area, musical culture was centered at and spread across churches. In order to appreciate Renaissance music, therefore, it is important to account for the influence of acoustics of churches at that era so that audiences today can experience homogeneous musical appreciation. We used an Active Field Control system to create the acoustics using measured the impulse responses (IRs) of a church. The system consists of directional microphones, head amps, a convolution engine, a matrix processor, amplifiers, and loudspeakers. And it picks up the direct response of performance, convolves it with the measured IRs, and reproduces the resulting sound using loudspeakers around the room. Loudspeaker positions in the performance hall are equivalent to the positions where the IRs had been measured at the church. This technique allows us to convincingly recreate not only the reverberation time but also the spatial impressions of the church at the performance hall. In addition, we modified the IRs so that the inherent acoustical characters of the performance space would less influence to recreation of the target church acoustics. We used a 6-channel recording/reproduction system to evaluate the modification to the IRs and the impressions of the recreated acoustics in the room. This paper presents the results of the experiment.

Discovering Your Inner Bat: Echo–Acoustic Target Ranging in Humans

Echolocation is typically associated with bats and toothed whales. To date, only few studies have investigated echolocation in humans. Moreover, these experiments were conducted with real objects in real rooms; a configuration in which features of both vocal emissions and perceptual cues are difficult to analyse and control. We investigated human sonar target-ranging in virtual echo-acoustic space, using a short-latency, real-time convolution engine. Subjects produced tongue clicks, which were picked up by a headset microphone, digitally delayed, convolved with individual head-related transfer functions and played back through earphones, thus simulating a reflecting surface at a specific range in front of the subject. In an adaptive 2-AFC paradigm, we measured the perceptual sensitivity to changes of the range for reference ranges of 1.7, 3.4 or 6.8 m. In a follow-up experiment, a second simulated surface at a lateral position and a fixed range was added, expected to act either as an interfering masker or a useful reference. The psychophysical data show that the subjects were well capable to discriminate differences in the range of a frontal reflector. The range-discrimination thresholds were typically below 1 m and, for a reference range of 1.7 m, they were typically below 0.5 m. Performance improved when a second reflector was introduced at a lateral angle of 45°. A detailed analysis of the tongue clicks showed that the subjects typically produced short, broadband palatal clicks with durations between 3 and 15 ms, and sound levels between 60 and 108 dB. Typically, the tongue clicks had relatively high peak frequencies around 6 to 8 kHz. Through the combination of highly controlled psychophysical experiments in virtual space and a detailed analysis of both the subjects' performance and their emitted tongue clicks, the current experiments provide insights into both vocal motor and sensory processes recruited by humans that aim to explore their environment by echolocation.

QCDNUM: Fast QCD evolution and convolution

The qcdnum program numerically solves the evolution equations for parton densities and fragmentation functions in perturbative QCD. Un-polarised parton densities can be evolved up to next-to-next-to-leading order in powers of the strong coupling constant, while polarised densities or fragmentation functions can be evolved up to next-to-leading order. Other types of evolution can be accessed by feeding alternative sets of evolution kernels into the program. A versatile convolution engine provides tools to compute parton luminosities, cross-sections in hadron–hadron scattering, and deep inelastic structure functions in the zero-mass scheme or in generalised mass schemes. Input to these calculations are either the qcdnum evolved densities, or those read in from an external parton density repository. Included in the software distribution are packages to calculate zero-mass structure functions in un-polarised deep inelastic scattering, and heavy flavour contributions to these structure functions in the fixed flavour number scheme. Program summary Program title: QCDNUM version: 17.00 Catalogue identifier: AEHV_v1_0 Program summary URL: http://cpc.cs.qub.ac.uk/summaries/AEHV_v1_0.html Program obtainable from: CPC Program Library, Queen's University, Belfast, N. Ireland Licensing provisions: GNU Public Licence No. of lines in distributed program, including test data, etc.: 45 736 No. of bytes in distributed program, including test data, etc.: 911 569 Distribution format: tar.gz Programming language: Fortran-77 Computer: All Operating system: All RAM: Typically 3 Mbytes Classification: 11.5 Nature of problem: Evolution of the strong coupling constant and parton densities, up to next-to-next-to-leading order in perturbative QCD. Computation of observable quantities by Mellin convolution of the evolved densities with partonic cross-sections. Solution method: Parametrisation of the parton densities as linear or quadratic splines on a discrete grid, and evolution of the spline coefficients by solving (coupled) triangular matrix equations with a forward substitution algorithm. Fast computation of convolution integrals as weighted sums of spline coefficients, with weights derived from user-given convolution kernels. Restrictions: Accuracy and speed are determined by the density of the evolution grid. Running time: Less than 10 ms on a 2 GHz Intel Core 2 Duo processor to evolve the gluon density and 12 quark densities at next-to-next-to-leading order over a large kinematic range.

Flexible hardware-friendly digital architecture for 2-D separable convolution-based scaling

There is not a single scaling technique that suits all kind of images. Final image quality (IQ) depends not only on the scale factor but also on the type of image (photo, CAD, Text...) the user is willing to print or display. Formally, any convolution-based scaling operation can be decomposed in three steps: an anti-aliasing filter, image reconstruction by continuous convolution and resampling to the final grid. Based on this formal framework, we propose a flexible hardware-friendly architecture to perform two-dimensional upscaling and downscaling at low hardware cost. In particular, we propose a discrete convolution engine operating a memory that stores a programmable 2-D-separable interpolation kernel. We also state a technique for optimizing the memory size given the kernel and the scale factor. Finally, we describe a novel flexible filter that overcomes aliasing artifacts regardless of image frequency content. The flexibility provided by the combination of the aforementioned elements allows the user to adjust the interpolation kernel and parameters to each specific type of image for IQ improvement.