Implementation Of Convolution Research Articles

Lightweight and Low-Parametric Network for Hardware Inference of Obstructive Sleep Apnea.

Background: Obstructive sleep apnea is a sleep disorder that is linked to many health complications and can even be lethal in its severe form. Overnight polysomnography is the gold standard for diagnosing apnea, which is expensive, time-consuming, and requires manual analysis by a sleep expert. Artificial intelligence (AI)-embedded wearable device as a portable and less intrusive monitoring system is a highly desired alternative to polysomnography. However, AI models often require substantial storage capacity and computational power for edge inference which makes it a challenging task to implement the models in hardware with memory and power constraints. Methods: This study demonstrates the implementation of depth-wise separable convolution (DSC) as a resource-efficient alternative to spatial convolution (SC) for real-time detection of apneic activity. Single lead electrocardiogram (ECG) and oxygen saturation (SpO2) signals were acquired from the PhysioNet databank. Using each type of convolution, three different models were developed using ECG, SpO2, and model fusion. For both types of convolutions, the fusion models outperformed the models built on individual signals across all the performance metrics. Results: Although the SC-based fusion model performed the best, the DSC-based fusion model was 9.4, 1.85, and 11.3 times more energy efficient than SC-based ECG, SpO2, and fusion models, respectively. Furthermore, the accuracy, precision, and specificity yielded by the DSC-based fusion model were comparable to those of the SC-based individual models (~95%, ~94%, and ~94%, respectively). Conclusions: DSC is commonly used in mobile vision tasks, but its potential in clinical applications for 1-D signals remains unexplored. While SC-based models outperform DSC in accuracy, the DSC-based model offers a more energy-efficient solution with acceptable performance, making it suitable for AI-embedded apnea detection systems.

Compiler Support for Sparse Tensor Convolutions

This paper extends prior work on sparse tensor algebra compilers to generate asymptotically efficient code for tensor expressions with affine subscript expressions. Our technique enables compiler support for a wide range of sparse computations, including sparse convolutions and pooling that are widely used in ML and graphics applications. We propose an approach that gradually rewrites compound subscript expressions to simple subscript expressions with loops that exploit the sparsity pattern of the input sparse tensors. As a result, the time complexity of the generated kernels is bounded by the number of stored elements and not by the shape of the tensors. Our approach seamlessly integrates into existing frameworks and is compatible with recent advances in compilers for sparse computations, including the flexibility to efficiently handle arbitrary combinations of different sparse tensor formats. The implementation of our algorithm is open source and upstreamed to the MLIR sparse compiler. Experimental results show that our method achieves 19.5x speedup when compared with the state-of-the-art compiler-based method at 99.9% sparsity. The generated sparse kernels start to outperform dense convolution implementations at about 80% sparsity.

Efficient Data-Driven Gap Filling of Satellite Image Time Series Using Deep Neural Networks with Partial Convolutions

Abstract The abundance of gaps in satellite image time series often complicates the application of deep learning models such as convolutional neural networks for spatiotemporal modeling. Based on previous work in computer vision on image inpainting, this paper shows how three-dimensional spatiotemporal partial convolutions can be used as layers in neural networks to fill gaps in satellite image time series. To evaluate the approach, we apply a U-Net-like model on incomplete image time series of quasi-global carbon monoxide observations from the Sentinel-5 Precursor (Sentinel-5P) satellite. Prediction errors were comparable to two considered statistical approaches while computation times for predictions were up to three orders of magnitude faster, making the approach applicable to process large amounts of satellite data. Partial convolutions can be added as layers to other types of neural networks, making it relatively easy to integrate with existing deep learning models. However, the approach does not provide prediction uncertainties and further research is needed to understand and improve model transferability. The implementation of spatiotemporal partial convolutions and the U-Net-like model is available as open-source software. Significance Statement Gaps in satellite-based measurements of atmospheric variables can make the application of complex analysis methods such as deep learning approaches difficult. The purpose of this study is to present and evaluate a purely data-driven method to fill incomplete satellite image time series. The application on atmospheric carbon monoxide data suggests that the method can achieve prediction errors comparable to other approaches with much lower computation times. Results highlight that the method is promising for larger datasets but also that care must be taken to avoid extrapolation. Future studies may integrate the approach into more complex deep learning models for understanding spatiotemporal dynamics from incomplete data.

Implementation of High Performance Convolution Based on Novel Mux-Multiplier

Implementation of High Performance Convolution Based on Novel Mux-Multiplier

Redundancy-free integrated optical convolver for optical neural networks based on arrayed waveguide grating.

Optical neural networks (ONNs) have gained significant attention due to their potential for high-speed and energy-efficient computation in artificial intelligence. The implementation of optical convolutions plays a vital role in ONNs, as they are fundamental operations within neural network architectures. However, state-of-the-art convolution architectures often suffer from redundant inputs, leading to substantial resource waste. Here, we demonstrate an integrated optical convolution architecture that leverages the inherent routing principles of arrayed waveguide grating (AWG) to execute the sliding of convolution kernel and summation of results. M × N multiply-accumulate (MAC) operations are facilitated by M + N units within a single clock cycle, thus eliminating the redundancy. In the experiment, we achieved 5 bit precision and 91.9 % accuracy in the handwritten digit recognition task confirming the reliability of our approach. Its redundancy-free architecture, low power consumption, high compute density (8.53 teraOP mm-1 s-1) and scalability make it a valuable contribution to the field of optical neural networks, thereby paving the way for future advancements in high-performance computing and artificial intelligence applications.

Design and Implementation of Deep Learning 2D Convolutions on Modern CPUs

Design and Implementation of Deep Learning 2D Convolutions on Modern CPUs

High Throughput Acceleration of Scabbard Key Exchange and Key Encapsulation Mechanism Using Tensor Core on GPU for IoT Applications

High throughput key encapsulations and decapsulations are needed by IoT applications in order to simultaneously process a multitude of small data in secure communication. In this paper, we present two novel techniques for accelerating the implementation of polynomial convolution on a GPU, utilizing advanced Tensor cores, which benefit the performance of key encapsulations. First, a polynomial re-structuring technique is proposed to allow several polynomials with distinct public keys to be processed in a single communication cycle. This is an improvement compared to the previous work by Lee et al. Next, we observe that polynomial convolution in some key encapsulation mechanisms contains reduction patterns that are not friendly to parallel implementation. We propose separating the multiplication and reduction processes so they can be parallelized independently. To verify the effectiveness of our proposed techniques, we applied it to two key-encapsulation mechanisms from the Scabbard post-quantum key-encapsulation mechanism suite and evaluate their performance. Experimental results show that polynomial convolution using Tensor cores is 1.05× faster (for the Florete scheme) and 3.6× faster (for the Sable scheme) than using CUDA core-based multiplication with conventional cores on a GPU. The Tensor cores based encapsulations and decapsulations are faster than a reference implementation on a CPU supporting AVX2 by more than 5.6× and 6.4×, respectively, for the Florete scheme and 8.3× and 13.3× faster, respectively, for the Sable scheme. This shows that the proposed techniques can achieve significantly higher throughput for key exchange and encapsulation mechanisms, which are important for securing IoT applications.

A descriptive behavior intention inference framework using spatio-temporal semantic features for human–robot real-time interaction

Visual behavior intention inference is crucial for enabling escort robots to interact naturally with humans, which is very challenging due to the big inner-class similarity and the small intra-class distinguishability of successive actions in the assistive scenario. To attain a reliable behavior intention inference, not only the current state of behaviors is concerned, but also the semantic information in both spatial and temporal domains plays an important role. This paper presents a segmentation–detection–recognition hierarchical system to represent the spatio-temporal semantic features for formulating descriptions of body parts, trajectories and deep relationships of sub-behaviors. Specifically, a dense trajectory matching scheme based on temporal sampling and Binarized Normed Gradients (BING) algorithm is formulated to segment the 3-Dimensional (3D) behavior cubes, based on which, local trajectories are obtained by clustering dense trajectories according to the distance similarity, and the body parts are then detected by multi-kernel learning of the encoded local features. Moreover, a global three-stream context Convolutional Neural Networks (CNN) is proposed for behavior classification by designing a texture module using expansion, connection and 1D convolution implementations. Based on transfer learning, scene information is also recognized efficiently. Finally, the semantic descriptors are modeled by two cascaded And-Or Graphs (AoGs) constraining the spatial scenarios and temporal sequences. Our unified approach is demonstrated on two public benchmarks containing long-term activities and on an escort robot for real-world applications.

A hardware-independent time estimation method for inference process of convolutional layers on GPU

Nowadays, various AI applications based on Convolutional Neural Networks (CNNs) are widely deployed on GPU-accelerated devices. However, due to the lack of visibility into GPU internal scheduling, accurately modeling the performance of CNN inference tasks or estimating the latency of CNN tasks that are executing or waiting on the GPU is challenging. This hurts the multi-model scheduling on multi-device and CNN real-time inference. Therefore, in this paper, we propose a time estimation method to estimate the forward execution time of a convolutional layer with an arbitrary shape on a GPU. The proposed method divides an explicit General Matrix Multiplication (GEMM) convolution operation into a series of estimatable GPU operations and constructs performance models at the level of sub-operations rather than hardware instructions or entire models. Also, the proposed method can be easily adapted to different hardware devices or underlying algorithm implementations, since it focuses on the variation of execution time relative to the input data scale, without focusing on specific instructions or hardware actions. According to the experiments on four typical CUDA compatible platforms, the proposed method has an average error rate of less than 5% for convolutional layers in some practical CNN models, and has about 8% error rate in estimating GEMM convolution implementations provided by cuDNN library. Experiments show that the proposed method can predict the forward execution time of convolutional layers of arbitrary size in CNN inference tasks on different GPU models.

Convolutional Neural Network Reference for Track-Before-Detect Applications

TBD (Track-Before-Detect) algorithms allow the detection and tracking of objects of which the signal is lost in the background noise. The use of convolutional neural networks (ConvNN) allows to obtain more effective algorithms than the previous, because it is possible to take into account the background as well as the spatial and temporal characteristics of the tracked object signal. The article presents solutions for taking into account the motion with variable trajectory and speed through segmental interpolation and rectification of the trajectory, which allows the effective convolutional implementation of the TBD algorithm. The boundary of object detection was determined depending on the number of pixels of the object in relation to the number of pixels of the image stack and signal strength for the simplest neural network, so it is possible to analyse and compare more complex solutions with the proposed reference.

Single-stage object detector with local binary pattern for remote sensing images

ABSTRACT The objects in remote sensing images usually have complex background and appear anywhere in any direction, which poses challenges for deep learning-based object detection. In order to solve the above problems, we embed the local binary pattern (LBP) into deep learning-based object detection to facilitate performance improvements. Specifically, we implement thep LBP via depthwise separable convolution without any learnable parameters, which makes LBP highlight the object-related regions of feature map instead of original images. The convolutional implementation of LBP makes it easy to embed it in deep learning-based object detection, so that deep learning-based object detection can possess the ability of LBP to extract the texture information from complex background without having to learn. The gradients of the branch where LBP convolution is located are blocked and cannot be backpropagated according to proposed LBP convolution, so we propose an effective residual block, called LBPRes (Residual block with Local Binary Pattern) for normal training. To deal with the problem of object direction changes, we introduce the rotation invariance into the LBPRes, denoted as RILBPRes (LBPRes with Rotation Invariance), which makes deep learning-based object detection have the local rotation invariance to cope with the object direction changes. Finally, we construct a single-stage object detection network S2LBPNet (Single Stage object detection Network with Local Binary Pattern) with a local binary pattern and conduct related experiments on the DIOR and HRRSD datasets. According to the experimental results, with a small number of parameters added, S2LBPNet outperforms the YOLOv5s baseline by 2.2% and 5.1% on DIOR and HRRSD datasets, respectively, which prove the superiority of the proposed S2LBPNet.

Context Adaptive Network for Image Inpainting.

In a typical image inpainting task, the location and shape of the damaged or masked area is often random and irregular. The vanilla convolutions widely used in learning-based inpainting models treat all spatial features as valid and share parameters across regions, making it difficult for them to cope with those irregular damages, and models tend to produce inpainting results with color discrepancy and blurriness. In this paper, we propose a novel Context Adaptive Network (CANet) to address this issue. The main idea of the proposed CANet is able to generate different weights depending on the miscellaneous input, which may help to complement images with multiple broken forms in a flexible way. Specifically, the proposed CANet has two novel context adaptive modules, namely, the context adaptive block (CAB) and the cross-scale contextual attention (CSCA), which utilize attention mechanisms to cope with diverse content breakdowns. The proposed CAB, during the forward propagation, uses an adaptive term to determine the importance between adaptive term and convolution kernel, so as to dynamically balance features based on the degree of breakage (confidence level or soft mask), and the overall calculation is formulated as a classic convolution implementation with an additional attention term to describe local structure. Besides, the proposed CSCA, not only takes advantage of the contextual attention module, but also considers cross-scale information transfer to generate reasonable features for damaged areas, thus alleviating the inefficiency of the long-range modeling capability of convolutional neural networks. Qualitative and quantitative experiments show that our method performs better than state-of-the-arts, producing clearer, more coherent and visually plausible inpainting results. The code can be found at github.com/dengyecode/CANet_image_inpainting.

A Lightweight Deep Learning Architecture for Vegetation Segmentation using UAV-captured Aerial Images

The unmanned aerial vehicle (UAV)-captured panoptic remote sensing images have great potential to promote robotics-inspired intelligent solutions for land cover mapping, disaster management, smart agriculture through automatic vegetation detection, and real-time environmental surveillance. However, many of these applications require faster execution of the tasks to get the job done in real-time. In this regard, this article proposes a lightweight convolutional neural network (CNN) architecture, also known as LW-AerialSegNet, that helps preserve the network’s feed-forward nature by increasing the intermediate layers to gather more crucial features for the segmentation task. Moreover, the network uses the concept of densely connected architecture and depth-wise separable convolution mechanisms to reduce the number of parameters of the model that can be deployed in the internet of things (IoT) edge devices to perform real-time segmentation. A UAV-based image segmentation dataset NITRDrone Dataset and Urban Drone dataset (UDD) are used to evaluate the proposed architecture. It has achieved an intersection over union (IoU) of 82% and 71% on the NITRDrone datasets UDD, respectively, thereby illustrating its superiority among the considered state-of-the-art mechanisms. The experimental results indicate that the implementation of depth-wise separable convolutions helps reduce the number of trainable parameters significantly, making it suitable to be applied on edge-computing devices at a smaller scale. The proposed architecture can be deployed in real-life settings on a UAV to extract objects such as vegetation and road lines, hence can be used in mapping urban areas, agricultural lands, etc.

Reformulating the direct convolution for high-performance deep learning inference on ARM processors

We present two high-performance implementations of the convolution operator via the direct algorithm that outperform the so-called lowering approach based on the im2col transform plus the gemm kernel on an ARMv8-based processor. One of our methods presents the additional advantage of zero-memory overhead while the other employs an additional yet rather moderate workspace, substantially smaller than that required by the im2col+gemm solution. In contrast with a previous implementation of a similar zero-memory overhead direct convolution, this work exhibits the key advantage of preserving the conventional NHWC data layout for the input/output activations of the convolution layers.

Multithreaded Convolution Implementation Based on Block Methods

A multithreaded convolution implementation based on block algorithms is considered. Convolution is essentially the basis of many methods that solve the problem of determining the degree of similarity or independence of two processes, in other words, when it is necessary to determine the degree of correlation. The algorithm itself is executed with a significant delay, because for its execution it is necessary to accumulate the entire signal and then process it. The analysis showed that one of the possible ways to reduce time costs is a multithreaded implementation of convolution based on block algorithms. The article shows the main features of the convolution implementation by the overlap method with addition and the overlap method with addition, as well as numerical examples. The results obtained show that the application of these methods without the use of a window function leads to significant distortions in the signal spectrum. Based on the results of the analysis, a universal scheme for performing convolution based on multithreaded processing of an input data block is proposed. This allows to achieve a good compromise between computational complexity, system architecture, and time costs.

Design and Implementation of 2D Convolution on x86/x64 Processors

In this paper, a new method for accelerating the 2D direct Convolution operation on x86/x64 processors is presented. It includes efficient vectorization by using SIMD intrinsics, bit-twiddling optimizations, the optimization of the division operation, multi-threading using OpenMP, register blocking and the shortest possible bit-width value of the intermediate results. The proposed method, which is provided as open-source, is general and can be applied to other processor families too, e.g., Arm. The proposed method has been evaluated on two different multi-core Intel CPUs, by using twenty different image sizes, 8-bit integer computations and the most commonly used kernel sizes (3x3, 5x5, 7x7, 9x9). It achieves from <inline-formula><tex-math notation="LaTeX">$2.8\times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$40\times$</tex-math></inline-formula> speedup over the Intel IPP library (OpenCV GaussianBlur and Filter2D routines), from <inline-formula><tex-math notation="LaTeX">$105 \times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$400 \times$</tex-math></inline-formula> speedup over the gemm-based convolution method (by using Intel MKL int8 matrix multiplication routine), and from <inline-formula><tex-math notation="LaTeX">$8.5\times$</tex-math></inline-formula> to <inline-formula><tex-math notation="LaTeX">$618\times$</tex-math></inline-formula> speedup over the vslsConvExec Intel MKL direct convolution routine. The proposed method is superior as it achieves far fewer arithmetical and load/store instructions.

A new three-dimensional elastography using phase based shifted Fourier transform

Elastography is an imaging technique with the ability to determine low quantities of some of the mechanical properties of tissues. The aim of our research is to design a new 3D algorithm using the Shifted Fourier Transform (SFT) to perform a quasi-static elastography. Our innovative idea is implementation of a 3D convolution instead of using three 2D convulsions. At first, we collected the raw data from Abaqus engineering software in the form of breast tissue with a coefficient of elasticity of healthy tissue and tumor tissue with a coefficient of elasticity of tumor tissue. The primary raw data consists of a number of points with x, y and z specified for tumor and healthy breast tissue. At this step, we simulated the displacements in directions of x, y and z at each point of the prescribed tissues for 15 ​mm displacement of probe in –Y direction then we collected 1831 points for tumor and 4186 points for breast before and after pressure. After applying a novel reconstruction algorithm, we convolved all images with the 3D Gabor filters to obtain phases, represented displacements of the breast and tumor images for before and after pressure. To reach this goal, we designed a Gabor filter bank based on the dimensions of the input images in different scales, directions, and deviations. Using the 3D SFT, we calculated the displacements of the breast and tumor tissues followed by 3D elastogram representation of the images. Finally, we implemented a 2D analysis of SFT in order to investigate validation of the 3D SFT. In 2D algorithm, we used three two-dimensional convulsions in XY, YZ and XZ planes. The results obtained from the small displacements marked by circles, confirmed the accuracy of the 3D SFT algorithm. These areas of interest are the tumor areas in the 2D analysis.

Deep Learning-Based 2-D Frequency Estimation of Multiple Sinusoidals.

Frequency estimation of 2-D multicomponent sinusoidal signals is a fundamental issue in the statistical signal processing community that arises in various disciplines. In this article, we extend the DeepFreq model by modifying its network architecture and apply it to 2-D signals. We name the proposed framework 2-D ResFreq. Compared with the original DeepFreq framework, the 2-D convolutional implementation of the matched filtering module facilitates the transformation from time-domain signals to frequency-domain signals and reduces the number of network parameters. The additional upsampling layer and stacked residual blocks are designed to perform superresolution. Moreover, we introduce frequency amplitude information into the optimization function to improve the amplitude accuracy. After training, the signals in the test set are forward-mapped to 2-D accurate and high-resolution frequency representations. Frequency and amplitude estimation are achieved by measuring the locations and strengths of the spectral peaks. We conduct numerical experiments to demonstrate the superior performance of the proposed architecture in terms of its superresolution capability and estimation accuracy.

Optimizing small channel 3D convolution on GPU with tensor core

In many scenarios, particularly scientific AI applications, algorithm engineers widely adopt more complex convolution, e.g. 3D CNN, to improve the accuracy. Scientific AI applications with 3D-CNN, which tends to train with volumetric datasets, substantially increase the size of the input, which in turn potentially restricts the channel sizes (e.g. less than 64) under the constraints of limited device memory capacity. Since existing convolution implementations tend to split and parallelize computing the small channel convolution from channel dimension, they usually cannot fully exploit the performance of GPU accelerator, in particular that configured with the emerging tensor core.In this work, we target on enhancing the performance of small channel 3D convolution on the GPU platform configured with tensor cores. Our analysis shows that the channel size of convolution has a great effect on the performance of existing convolution implementations, that are memory-bound on tensor core. By leveraging the memory hierarchy characteristics and the WMMA API of tensor core, we propose and implement holistic optimizations for both promoting the data access efficiency and intensifying the utilization of computing units. Experiments show that our implementation can obtain 1.1x–5.4x speedup comparing to the cuDNN’s implementations for the 3D convolutions on different GPU platforms. We also evaluate our implementations on two practical scientific AI applications and observe up to 1.7x and 2.0x overall speedups compared with using cuDNN on V100 GPU.

Convolutional Implementation and Analysis of the CFS-PML ABC for the FDTD Method Based on Wave Equation

In this letter, the convolutional implementation of the complex frequency shifted perfectly matched layer (CFS-PML) absorbing boundary condition (ABC) is proposed for the finite-difference time-domain (FDTD) method of solving wave equation. The FDTD solution time-marching formulations of the developed wave-equation convolutional CFS-PML (CPML) ABC are derived. Several numerical experiments are used to verify the capacity of the proposed algorithm. Solution results confirm that the proposed CPML ABC can obtain excellent absorption for both traveling waves and evanescent waves.