Direct Convolution Research Articles

EcoFlow: Efficient Convolutional Dataflows on Low-Power Neural Network Accelerators

Dilated and transposed convolutions are widely used in modern convolutional neural networks (CNNs). These kernels are used extensively during CNN training and inference of applications such as image segmentation and high-resolution image generation. We find that commonly-used low-power CNN inference accelerators are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">not</i> optimized for both these convolutional kernels. Dilated and transposed convolutions introduce significant zero padding when mapped to the underlying spatial architecture, significantly degrading performance and energy efficiency. Existing approaches that address this issue require significant design changes to the otherwise simple, efficient, and well-adopted architectures used to compute direct convolutions. To address this challenge, we propose EcoFlow, a new set of dataflows and mapping algorithms for dilated and transposed convolutions. These algorithms are tailored to execute efficiently on existing low-cost, small-scale spatial architectures and requires minimal changes to existing accelerators. At its core, EcoFlow eliminates zero padding through careful dataflow orchestration and data mapping tailored to the spatial architecture. We evaluate EcoFlow on CNN training workloads and Generative Adversarial Network (GAN) workloads. Experiments in our new cycle-accurate simulator show that, using a common CNN inference accelerator, EcoFlow 1) reduces end-to-end CNN training time between 7-85%, and 2) improves end-to-end GAN training performance between 29-42%, compared to state-of-the-art CNN dataflows.

Optimizing Data Flow in Binary Neural Networks.

Binary neural networks (BNNs) can substantially accelerate a neural network's inference time by substituting its costly floating-point arithmetic with bit-wise operations. Nevertheless, state-of-the-art approaches reduce the efficiency of the data flow in the BNN layers by introducing intermediate conversions from 1 to 16/32 bits. We propose a novel training scheme, denoted as BNN-Clip, that can increase the parallelism and data flow of the BNN pipeline; specifically, we introduce a clipping block that reduces the data width from 32 bits to 8. Furthermore, we decrease the internal accumulator size of a binary layer, usually kept using 32 bits to prevent data overflow, with no accuracy loss. Moreover, we propose an optimization of the batch normalization layer that reduces latency and simplifies deployment. Finally, we present an optimized implementation of the binary direct convolution for ARM NEON instruction sets. Our experiments show a consistent inference latency speed-up (up to 1.3 and 2.4× compared to two state-of-the-art BNN frameworks) while reaching an accuracy comparable with state-of-the-art approaches on datasets like CIFAR-10, SVHN, and ImageNet.

Direct Optical Convolution Computing Based on Arrayed Waveguide Grating Router

AbstractOptical convolution computing is gaining traction owing to its inherent parallelism, multi‐dimensional processing, and energy efficiency. To handle input dimensions of N, conventional implementations necessitate N2 optical elements, such as Mach–Zehnder interferometers or micro‐ring resonators, to process multiply‐accumulate (MAC) operations, limiting scalability and resulting in elevated power consumption. Here, a direct convolution computing method based on wavelength routing, utilizing the unique sliding property of an arrayed waveguide grating router (AWGR) to perform the sliding window operation of the convolution in the wavelength–space domains is proposed. With two input vectors directly loaded onto two modulator arrays, the convolution result is instantaneously produced at a photodetector array. The entire convolution computation is executed within a single clock cycle without the need for preprocessing or decomposition into elementary MAC operations. The number of active elements is minimal, only needed for input/output. The proposed optical convolution unit has striking advantages of high scalability, high speed, and processing simplicity compared to those based on optical matrix‐vector multipliers. In the first experimental demonstration, a remarkable classification accuracy of up to 98.2% in handwritten digit recognition tasks using a LeNet‐5 neural network is achieved.

DETECTING THE SECURITY LEVEL OF VARIOUS CRYPTOSYSTEMS USING MACHINE LEARNING MODELS

In contemporary data management, cloud storage is widely adopted, yet it remains susceptible to security vulnerabilities. Utilizing cryptography techniques is essential for enhancing data security in such environments. One promising approach is hybrid cryptography, which combines multiple algorithms to bolster protection. Our proposed solution integrates RDH with DES algorithms, leveraging their respective strengths to fortify data security before it's stored in the cloud. Extensive testing has validated the efficacy of this approach. Specifically, we introduce an RDH with DES block-based transformation algorithm tailored for image content protection. Notably, our framework enables direct image retrieval and convolution on the content-protected images, enhancing usability alongside security. Keywords: deployment models, Infrastructure as a service, cryptosystems, machine learning

Comparing subjective similarity ratings and quantitative errors for the evaluation of free-field binaural panning techniques

A free-field sound source can be accurately rendered over headphones via direct convolution with head-related impulse responses (HRIRs). Using binaural panning techniques, a free-field source can be approximated, which can be thought of as HRIR reconstruction. However, a loss of fidelity occurs related to the panning algorithm’s spatial resolution. Previously, perceptual comparisons of noise bursts were made between panned sources and direct convolution with measured HRIRs. Reconstructions were generated using five different binaural panning methods, both with and without time-alignment of the HRIRs with separate application of the interaural time delay (ITD). To further explore the listening test results, the quantitative differences between the reconstructed and original HRIRs were investigated alongside the perceptual data. Several perceptually-motivated error metrics were evaluated, including errors in both ITD and interaural level difference. The fidelity of the reconstructed HRTF magnitude response was evaluated using metrics that leveraged auditory modeling steps. Both subjective and objective results support that HRIR time-alignment reduces the number of filters needed for high perceptual accuracy. A principal component-base rendering filter set produced the best subjective accuracy with only a small number of required filters (16-25). The ability to predict perceptually detectable degradation via quantitative errors will be discussed.

Modification and Comparison of Methods for Predicting the Moisture Content of Dead Fuel on the Surface of Quercus mongolica and Pinus sylvestris var. mongolica under Rainfall Conditions

The surface fine dead fuel moisture content (FFMC) is an important factor in predicting forest fire risk and is influenced by various meteorological factors. Many prediction methods rely on temperature and humidity as factors, resulting in poor model prediction accuracy under rainfall conditions. At the same time, there is an increasing number of methods based on machine learning, but there is still a lack of comparison with traditional models. Therefore, this paper selected the broad-leaved forest tree species Quercus mongolica and the coniferous forest species Pinus sylvestris var. mongolica in Northeast China. Taking surface dead fine fuel as the research object, we used indoor simulated rainfall experiments to explore the impact of rainfall on the surface dead fuel moisture content. The prediction model for surface dead fuel moisture content was modified by the direct estimation method. Finally, using field data, the direct estimation method and convolution neural network (CNN) model were used in the comparison. The rainfall simulation results showed that the indoor fuel moisture content had a logarithmic increasing trend. Rainfall and previous fuel moisture content had a significant impact on the fuel moisture content prediction model, and both the relational model and nonlinear model performed well in predicting fuel moisture content under indoor rainfall conditions. Under field conditions, humidity, temperature and rainfall played a significant role in fuel moisture content. Compared with the unmodified direct estimation method, the modified direct estimation method significantly improved the prediction accuracy and the goodness of fit (R2) increased from 0.85–0.94 to 0.94–0.96. Mean absolute error (MAE) decreased from 9.18–18.33% to 6.86–10.74%, and mean relative error (MRE) decreased from 3.97–17.18% to 3.53–14.48%. The modified direct estimation method has higher prediction accuracy compared with the convolutional neural network model; the R2 value was above 0.90, MAE was below 8.11%, and MRE was below 8.87%. The modified direct estimation method had the best prediction effect among them. This study has a certain reference value for the prediction model of surface fuel moisture content in post-rainfall fire risk assessment and is also of great significance for forest fire management in Northeast China.

Features of the trajectory signal processing in the distributed radar when rotating the joint set of transmitting and receiving antennas

Increasing the resolution of ground survey radar systems (radars) makes it possible to obtain images of ground objects and the observed surface with high quality. At the same time, as it is known, the most difficult is to provide high resolution radar by angular coordinate. The method based on synthesis of artificial aperture allows to increase significantly the angular resolution of airborne radar. The capabilities of a radar station with synthesis of antenna aperture (SAR) in angular resolution are determined by the structure of the spatial and temporal trajectory signal formed in the process of synthesis. In this case, the resolution is determined by the angular size of the synthesized aperture. The purpose of this article is to consider the features of the trajectory signal structure and its processing during synthesis of an artificial antenna aperture in a distributed radar station consisting of a set of two receiving and one receiving-transmitting modules rotating in a circle. It is required to analyze the trajectory signal structure and evaluate the potentially achievable angular resolution capabilities of a distributed radar station when synthesizing an artificial antenna aperture under conditions of rotation of a spatially spaced set of phase centers of real antennas. To analyze the structure of the trajectory signal generated in a distributed radar station, we have considered the geometry of the problem, because it determines the change in the current distance between the phase center of the antennas and the observation object and is the main generating factor of the arising modulation of this signal. An informative tool for analyzing the characteristics of radar systems with synthesis of antenna aperture by potential resolution, is the analysis of the output response of the optimal trajectory signal processing system. The direct convolution method has been used to obtain the responses. Evaluation of the efficiency of coherent processing of the trajectory signals formed at the separated reception points has been carried out by computer simulation. From the simulation results presented in the figures in the article, it can be seen that the coherent processing of the trajectory signal formed at the separated receiving points during rotation of the phase center of real antennas, allows to increase the azimuth resolution of the radar. Thus, the results presented in the article show that rotation of the separated phase centers of the antennas of the receiving and receiving and transmitting modules within a range of up to 120 degrees provides the formation of a synthesized aperture when processing the trajectory signal from the observed target at each receiving point. The best azimuth resolution is obtained by processing the trajectory signal at the location of the transceiver module antenna. When coherent processing of trajectory signals formed at separated receiving points is performed together, the azimuth resolution of the radar station increases by about two times compared to the case of their separate handling.

Micromechanical modeling for the prediction of time-dependent effective properties and macroscopic coupled field response of polymer reinforced piezoelectric composites

In the time domain, a new straightforward micromechanical modeling is developed to predict the time-dependent behavior and the macroscopic coupled response of polymer piezoelectric composite materials. The modeling is based on the dynamic electroelastic Green’s tensor allowing the derivation of convolution localization equations in Stieltjes space. The Mori-Tanaka micromechanical model is adopted to derivate the dynamic electroelastic localization tensor. The presented modeling leads to an expression of the time-dependent effective electroelastic properties directly in the time domain, without an iteration procedure, through convolution products in Stieltjes space. It allows the prediction of the time-dependent behavior and the coupled macroscopic response while taking into account the constituent properties, volume fractions, and shape of inclusions. In implementing the expression of the effective behavior, one has to go through the computation of direct and inverse tensorial convolution products. A trapezoidal numerical scheme is adopted for this sake. Many numerical results are presented, and for the sake of validation, a comparison is made with the ones obtained based on the Laplace transform approach as well as with experimental and numerical results.

Free-field perceptual evaluation of virtual acoustic rendering algorithms using two head-related impulse response delay treatment strategies

Sound scenes can be auralized over headphones using binaural rendering techniques in conjunction with a set of head-related impulse responses (HRIRs). If the directions of the sound objects to be rendered are known, either virtual loudspeaker or Ambisonic scene-based techniques may be used, each of which introduce spatial and timbral artifacts at lower spatial resolutions. Neal and Zahorik quantitatively evaluated the effect of separately applying the HRIR delays to time-aligned HRIRs for use with virtual loudspeaker array techniques, referred to as the prior HRIR delays treatment strategy (J. Acoust. Soc. Am. 2022). The present work aims to perceptually validate their quantitative results in a listening study. Free-field point sources were binaurally rendered using five different methods: vector-based amplitude panning, Ambisonics panning, direct spherical harmonic transform of the HRIR set, MagLS, and Principal Component-Base Amplitude Panning (Neal and Zahorik, Audio Eng. Soc. AVAR Conference 2022). Renderings were simulated at various directions and spatial resolutions, both with and without the prior HRIR delays treatment strategy and compared to direct HRIR convolution. Broadband noise served as a critical test signal for revealing changes in timbre and localization and a high-resolution binaural head HRIR dataset was used. The subjective data analysis will be presented.

Fault prediction using fuzzy convolution neural network on IoT environment with heterogeneous sensing data fusion

Using multi-source sensing data based on the Internet of things and fusion in conjunction with fuzzy convolutional neural networks to classify and predict mechanical failures has emerged as a very pertinent area of research. This study presents a fuzzy convolutional neural network-based technique to extract and diagnose the defect indicators utilizing the bearing dataset from Case Western Reserve University (CWRU). In addition, the proposed model's effectiveness is evaluated with the help of a model based on a direct convolution neural network. The strategy based on Fuzzification yielded superior results when applied to a fully linked layer for classification. When applied to the bearing dataset, the proposed approach produces an average classification accuracy of 99.87% using the fuzzy layer. The proposed method is evaluated compared to other methods based on machine learning and deep learning. The accuracy achieved using the proposed method is superior to that achieved using the existing approaches in every imaginable working environment. The proposed approach can be used to diagnose mechanical motor faults and provide earlier recommendations.

Geodesic Convolutional Neural Network Characterization of Macro-Porous Latent Thermal Energy Storage

Abstract High-temperature latent heat thermal energy storage with metallic alloy phase change materials (PCMs) utilize the high latent heat and high thermal conductivity to gain a competitive edge over existing sensible and latent storage technologies. Novel macroporous latent heat storage units can be used to enhance the limiting convective heat transfer between the heat transfer fluid and the PCM to attain higher power density while maintaining high energy density. 3D monolithic percolating macroporous latent heat storage unit cells with random and ordered substructure topology were created using synthetic tomography data. Full 3D thermal computational fluid dynamics (CFD) simulations with phase change modeling was performed on 1000+ such structures using effective heat capacity method and temperature- and phase-dependent thermophysical properties. Design parameters, including transient thermal and flow characteristics, phase change time and pressure drop, were extracted as output scalars from the simulated charging process. As such structures cannot be parametrized meaningfully, a mesh-based Geodesic Convolutional Neural Network (GCNN) designed to perform direct convolutions on the surface and volume meshes of the macroporous structures was trained to predict the output scalars along with pressure, temperature, velocity distributions in the volume, and surface distributions of heat flux and shear stress. An Artificial Neural Network (ANN) using macroscopic properties—porosity, surface area, and two-point surface-void correlation functions—of the structures as inputs was used as a standard regressor for comparison. The GCNN exhibited high prediction accuracy of the scalars, outperforming the ANN and linear/exponential fits, owing to the disentangling property of GCNNs where predictions were improved by the introduction of correlated surface and volume fields. The trained GCNN behaves as a coupled CFD-heat transfer emulator predicting the volumetric distribution of temperature, pressure, velocity fields, and heat flux and shear stress distributions at the PCM–HTF interface. This deep learning based methodology offers a unique, generalized, and computationally competitive way to quickly predict phase change behavior of high power density macroporous structures in a few seconds and has the potential to optimize the percolating macroporous unit cells to application specific requirements.

Analysis and Optimization of Direct Convolution Execution on Multi-Core Processors

Nowadays, convolutional neural networks are among the most widely used types of deep learning networks thanks to their usefulness in many application domains. There are many efforts to find methods to increase their training and inference performance and efficiency. One of the most widely used technique to implement convolution consists of flattening tensors into 2D matrices and carrying out the operation through a matrix-matrix multiplication routine, which has highly optimized implementations in high-performance libraries. However, this kind of approach uses extra time and memory to transform and store the tensors involved. For this reason, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">direct convolution</i> is becoming increasingly popular. Direct convolution can be implemented as a series of nested loops iterating over tensor dimensions and it does not require extra memory. In this work, we evaluate on various multi-core CPUs the performance and scalability effects deriving from different parallelization strategies, loop organizations, and SIMD-vectorization approaches with different compilers in relation with architectural aspects. We discuss each parameter thoroughly and distill our findings in a set of heuristics that can be used to quickly achieve a high-performance implementation in accordance to the underlying hardware and the characteristics of the convolutional layer at hand. By adopting a per-layer approach, we increase performance up to 60-70% compared to a static implementation for all the layers. Moreover, our results are comparable, or even better (up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.67\times $ </tex-math></inline-formula> speedup) than matrix-matrix multiplication-based convolution in a multi-core system.

Novel Convolutions for Semantic Segmentation of Remote Sensing Images

The networks are required to be capable of learning low-level features well when applied to remote sensing image semantic segmentation task. To capture accurate and abundant low-level semantic information, the early feature extractor layer is crucial to the whole network, because all the subsequent features are inferred from that base. To address the low-level feature extraction issue and overcome the shortcomings of traditional convolution like too many parameters or limited receptive field, some novel convolution units have been proposed in the literature. In this paper, we propose two elaborately designed and portable yet effective convolution units, i.e., directional convolution (DC) and large field convolution (LFC), combined as the extractor of low-level semantic features. DC is designed to extract directional features from specific directions, and LFC can achieve a large receptive field with few parameters. Experiment results on two public datasets provide evidence that our convolution units can help deep learning networks improve performance stably and comprehensively compared to the baseline networks.

ECBC: Efficient Convolution via Blocked Columnizing.

Direct convolution methods are now drawing increasing attention as they eliminate the additional storage demand required by indirect convolution algorithms (i.e., the transformed matrix generated by the im2col convolution algorithm). Nevertheless, the direct methods require special input-output tensor formatting, leading to extra time and memory consumption to get the desired data layout. In this article, we show that indirect convolution, if implemented properly, is able to achieve high computation performance with the help of highly optimized subroutines in matrix multiplication while avoid incurring substantial memory overhead. The proposed algorithm is called efficient convolution via blocked columnizing (ECBC). Inspired by the im2col convolution algorithm and the block algorithm of general matrix-to-matrix multiplication, we propose to conduct the convolution computation blockwisely. As a result, the tensor-to-matrix transformation process (e.g., the im2col operation) can also be done in a blockwise manner so that it only requires a small block of memory as small as the data block. Extensive experiments on various platforms and networks validate the effectiveness of ECBC, as well as the superiority of our proposed method against a set of widely used industrial-level convolution algorithms.

FPGA-Based Design for Accelerating 3D Convolutional Neural Networks

Three-dimensional convolutional neural networks (3D CNNs) have been shown to be excellent for classification in human action recognition, hyperspectral image classification and many other aspects. However, 3D CNNs often contain convolution kernels of different sizes and a large number of operations, and the numerous operations make 3D CNNs consume a lot of time when executing inference on resource-constrained terminals. Aiming at these two points, this paper proposes a 3D CNN accelerator design based on loop optimization and weight reuse. The design adopts loop tiling, loop unrolling and loop fusion to optimize the direct convolution form of the 3D convolution operation; and guided by the optimized form of the 3D convolution operation, a module that can accelerate different 3D convolution operations is designed; and a memory access method based on weight reuse is also designed to reduce the memory access time of features and weights. Experimental results show that the proposed accelerator not only supports convolution kernels of different sizes, but also has a performance density of 1.59, which outperforms most existing accelerators.

Reformulating the direct convolution for high-performance deep learning inference on ARM processors

We present two high-performance implementations of the convolution operator via the direct algorithm that outperform the so-called lowering approach based on the im2col transform plus the gemm kernel on an ARMv8-based processor. One of our methods presents the additional advantage of zero-memory overhead while the other employs an additional yet rather moderate workspace, substantially smaller than that required by the im2col+gemm solution. In contrast with a previous implementation of a similar zero-memory overhead direct convolution, this work exhibits the key advantage of preserving the conventional NHWC data layout for the input/output activations of the convolution layers.

Design and Implementation of 2D Convolution on x86/x64 Processors

In this paper, a new method for accelerating the 2D direct Convolution operation on x86/x64 processors is presented. It includes efficient vectorization by using SIMD intrinsics, bit-twiddling optimizations, the optimization of the division operation, multi-threading using OpenMP, register blocking and the shortest possible bit-width value of the intermediate results. The proposed method, which is provided as open-source, is general and can be applied to other processor families too, e.g., Arm. The proposed method has been evaluated on two different multi-core Intel CPUs, by using twenty different image sizes, 8-bit integer computations and the most commonly used kernel sizes (3x3, 5x5, 7x7, 9x9). It achieves from <inline-formula><tex-math notation="LaTeX">$2.8\times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$40\times$</tex-math></inline-formula> speedup over the Intel IPP library (OpenCV GaussianBlur and Filter2D routines), from <inline-formula><tex-math notation="LaTeX">$105 \times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$400 \times$</tex-math></inline-formula> speedup over the gemm-based convolution method (by using Intel MKL int8 matrix multiplication routine), and from <inline-formula><tex-math notation="LaTeX">$8.5\times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$618\times$</tex-math></inline-formula> speedup over the vslsConvExec Intel MKL direct convolution routine. The proposed method is superior as it achieves far fewer arithmetical and load/store instructions.

Enhanced Spatial-Temporal Salience for Cross-View Gait Recognition

Gait recognition can be used in person identification and re-identification by itself or in conjunction with other biometrics. Although gait has both spatial and temporal attributes, and it has been observed that decoupling spatial feature and temporal feature can better exploit the gait feature on the fine-grained level. However, the spatial-temporal correlations of gait video signals are also lost in the decoupling process. Direct 3D convolution approaches can retain such correlations, but they also introduce unnecessary interferences. Instead of common 3D convolution solutions, this paper proposes an integration of decoupling process into a 3D convolution framework for cross-view gait recognition. In particular, a novel block consisting of a Parallel-insight Convolution layer integrated with a Spatial-Temporal Dual-Attention (STDA) unit is proposed as the basic block for global spatial-temporal information extraction. Under the guidance of the STDA unit, this block can well integrate spatial-temporal information extracted by two decoupled models and at the same time retain the spatial-temporal correlations. In addition, a Multi-Scale Salient Feature Extractor is proposed to further exploit the fine-grained features through context awareness extension of part-based features and adaptively aggregating the spatial features. Extensive experiments on three popular gait datasets, namely CASIA-B, OULP and OUMVLP, demonstrate that the proposed method outperforms state-of-the-art methods.

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

Mapping of gold-related alteration minerals and linear structures using ASTER data in the Giyani Greenstone Belt, South Africa

This paper focuses on the mapping of alteration minerals and linear structures associated with gold mineralization in the Giyani Greenstone Belt (GGB). Spectral Information Divergence (SID) and Linear Spectral Unmixing (LSU) classification algorithms were applied to the ASTER data using image extracted endmember spectra which match the spectral profiles of predominant alteration minerals (biotite and calcite) related to gold mineralization in the GGB. Directional convolution filters were also applied to the ASTER data to extract structural lineaments that may be related to gold mineralization. In addition, Fry analysis of known gold occurrences in the GGB was conducted to determine the major distribution patterns along which gold mineralization have occurred. LSU and SID results show several pixels classified as alteration minerals corresponding to known gold occurrences. SID results also show minerals overlapping into the gneissic rock, which is a rare host of gold mineralization in the study area. On the other hand, LSU results show high abundances of calcite and biotite generally confined within the mafic-ultramafic greenstone rocks, which have been hypothesized as the source of gold-mineralized fluids in the GGB. In general, LSU show distinct broad regions classified as alteration minerals corresponding to known gold occurrences that have been previously reported to host these alteration minerals. Directional convolution filters enabled the extraction of predominately NE-SW oriented linear structures, including the major shear zone which is associated with gold mineralization in the GGB. Fry analysis revealed three major distribution patterns: N-S, NE-SW, ENE-WSW along which gold mineralization have occurred. These patterns correspond to major lineaments associated with gold mineralization reported in the previous structural studies of the GGB.