2D Convolution Operation Research Articles

Context-based local-global fusion network for 3D point cloud classification and segmentation

3D point clouds have gained much research attention because of their ability to represent the spatial information of real-world environments in a detailed manner. Despite recent progress in point cloud processing with deep neural networks, most of them either implement sophisticated local feature aggregation methods or imitate 2D convolution operations in the range of K nearest neighbors with limited local context information. These methods may struggle to distinguish between similar geometric shapes within the local region of K nearest neighbors, such as doors and walls. To address this issue, we propose a novel local–global fusion network that captures the diverse local geometric shapes with global structure information. The proposed local–global fusion network comprises two main modules. Firstly, we have developed an effective approach for local context learning using incremental dilated KNN (IDKNN) as the neighbor selecting mechanism to enlarge the receptive field and incorporate more reliable points for local geometric shape learning. Secondly, a three-direction region-wise spatial attention (TRSA) algorithm has been developed to explore the global contextual dependencies. For global context learning, we first split the entire 3D space into regions with equal numbers of points, and, then, intra-region context features are extracted to learn the inter-region relations from three orthogonal directions, taking global structural knowledge into account. By fusing the local context information and global contextual dependencies, we establish a Local-Global Fusion Network, end-to-end framework, called LGFNet. Extensive experimental results on several benchmark datasets clearly demonstrate our approach can achieve state-of-the-art (SOTA) performance on point cloud classification, part segmentation, and indoor semantic segmentation. In addition, TRSA and IKDNN can be easily used in a plug-and-play fashion with various existing SOTA networks to substantially improve their performance. Our code is available at https://github.com/jasonwjw/IDKNN

Spatiotemporal Fusion Model of Remote Sensing Images Combining Single-Band and Multi-Band Prediction

In recent years, convolutional neural network (CNN)-based spatiotemporal fusion (STF) models for remote sensing images have made significant progress. However, existing STF models may suffer from two main drawbacks. Firstly, multi-band prediction often generates a hybrid feature representation that includes information from all bands. This blending of features can lead to the loss or blurring of high-frequency details, making it challenging to reconstruct multi-spectral remote sensing images with significant spectral differences between bands. Another challenge in many STF models is the limited preservation of spectral information during 2D convolution operations. Combining all input channels’ convolution results into a single-channel output feature map can lead to the degradation of spectral dimension information. To address these issues and to strike a balance between avoiding hybrid features and fully utilizing spectral information, we propose a remote sensing image STF model that combines single-band and multi-band prediction (SMSTFM). The SMSTFM initially performs single-band prediction, generating separate predicted images for each band, which are then stacked together to form a preliminary fused image. Subsequently, the multi-band prediction module leverages the spectral dimension information of the input images to further enhance the preliminary predictions. We employ the modern ConvNeXt convolutional module as the primary feature extraction component. During the multi-band prediction phase, we enhance the spatial and channel information captures by replacing the 2D convolutions within ConvNeXt with 3D convolutions. In the experimental section, we evaluate our proposed algorithm on two public datasets with 16x resolution differences and one dataset with a 3x resolution difference. The results demonstrate that our SMSTFM achieves state-of-the-art performance on these datasets and is proven effective and reasonable through ablation studies.

Two-stage multi-dimensional convolutional stacked autoencoder network model for hyperspectral images classification

Deep learning models have been widely used in hyperspectral images classification. However, the classification results are not satisfactory when the number of training samples is small. Focused on above-mentioned problem, a novel Two-stage Multi-dimensional Convolutional Stacked Autoencoder (TMC-SAE) model is proposed for hyperspectral images classification. The proposed model is composed of two sub-models SAE-1 and SAE-2. The SAE-1 is a 1D autoencoder with asymmetric structre based on full connection layers and 1D convolution layers to reduce spectral dimensionality. The SAE-2 is a hybrid autoencoder composed of 2D and 3D convolution operations to extract spectral-spatial features from the reduced dimensionality data by SAE-1. The SAE-1 is trained with raw data by unsupervised learning and the encoder of SAE-1 is employed to reduce spectral dimensionality of raw data. The data after dimension reduction is used to train the SAE-2 by unsupervised learning. The fine-tuning of SAE-2 encoder and the training of classifier are implemented simultaneously with small number of samples by supervised learning. Comparative experiments are performed on three widely used hyperspectral remote sensing data. The extensive comparative experiments demonstrate that the proposed architecture can effectively extract deep features and maintain high classification accuracy with small number of training samples.

A Bandwidth-Efficient High-Performance RTL-Microarchitecture of 2D-Convolution for Deep Neural Networks

The computation complexity and huge memory access bandwidth of the convolutional layers in convolutional neural networks (CNNs) require specialized hardware architectures to accelerate CNN’s computations while keeping hardware costs reasonable for area-constrained embedded applications. This paper presents an RTL (Register Transfer Logic) level microarchitecture of hardware- and bandwidth-efficient high-performance 2D convolution unit for CNN in deep learning. The 2D convolution unit is made up of three main components including a dedicated Loader, a Circle Buffer, and a MAC (Multiplier-Accumulator) unit. The 2D convolution unit has a 2-stage pipeline structure that reduces latency, increases processing throughput, and reduces power consumption. The architecture proposed in the paper eliminates the reloading of both the weights as well as the input image data. The 2D convolution unit is configurable to support 2D convolution operations with different sizes of input image matrix and kernel filter. The architecture can reduce memory access time and power as well as execution time thanks to the efficient reuse of the preloaded input data, while simplifying hardware implementation. The 2D convolution unit has been simulated and implemented on Xilinx's FPGA platform to evaluate its superiority. Experimental results show that our design is 1.54× and 13.6× faster in performance than the design in [7] and [8], respectively, at lower hardware cost without using any FPGA’s dedicated hardware blocks. By reusing preloaded data, our design achieves a bandwidth reduction ratio between 66.4% and 90.5%.

Shuffled-Xception-DarkNet-53: A content-based image retrieval model based on deep learning algorithm

This paper proposes Shuffled-Xception-DarkNet-53, an advanced version of DarkNet-53 for Content-Based Image Retrieval (CBIR). The proposed model introduced the notion of the Shuffled -Xception module, which uses three sets of 1 × 1, 3 × 3, and 5 × 5 size filters using a serial connection in place of a single 3 × 3 size filter to extract more refined features from the input images. Instead of the standard 2D Convolution operation, 'Group Convolution' is employed in the proposed Shuffled -Xception module to make the training process of the proposed CNN more efficient. Furthermore, 'Group Convolution' improves the co-relations among the filters of the corresponding Shuffled -Xception module, resulting in more informative features. Between every two serial Group Convolution layers of the same size, one Channel Shuffle layer is used to prevent the loss of information flow among the channels of different groups. The proposed method outperformed the twelve compared methods, including conventional and CNN-based CBIR methods, in ten standard image datasets.

HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data

Privacy-preserving solutions enable companies to offload confidential data to third-party services while fulfilling their government regulations. To accomplish this, they leverage various cryptographic techniques such as Homomorphic Encryption (HE), which allows performing computation on encrypted data. Most HE schemes work in a SIMD fashion, and the data packing method can dramatically affect the running time and memory costs. Finding a packing method that leads to an optimal performant implementation is a hard task. We present a simple and intuitive framework that abstracts the packing decision for the user. We explain its underlying data structures and optimizer, and propose a novel algorithm for performing 2D convolution operations. We used this framework to implement an inference operation over an encrypted HE-friendly AlexNet neural network with large inputs, which runs in around five minutes, several orders of magnitude faster than other state-of-the-art non-interactive HE solutions.

Objects outline delineation by nonliniear filtration methods for boundary pixels

A processing of border pixel from pixel map originate problem while work with outline objects segmentation algorithm on bitmap images, which are based on 2D convolution operation. The problem correspond with addition set of pixels which are outside pixel matrix border of processing image. This problem is actual also in other algorithms, which have a common feature – 2D convolution operation. The level of importance are defined by multi used nonlinear filters to border pixels and then union a few images without distortions at the boundaries. The existing methods of pixel set redefining on pre processing step are primitive and affects on subsequent stages of processing. This affects exist between methods of redefining border pixels and algorithms of outline objects segmentation: Sharra, Pruitta, Sobel. The object of investigation is pixel map of halftone image. The subject of investigation is relation assessment between algorithms of outline objects segmentation with methods redefining border pixels. The main target is describing of features special methods to redefining border pixels which positively affecting the subsequent evaluation of the responses for this pixels by algorithms of outline objects segmentation. Analysis, the influence of the algorithm for the outline objects segmentation on the boundary pixels from the preliminary selection of methods of redefinition is carried out by example of combining several satellite images along the outline of urban infrastructure. For simplicity and by necessary of large data volume visualization (pixel half tone matrix size) presented graphs: halftone codes for redefining and border pixels, pixels requests for the algorithm of outline objects segmentation. By analysis criteria and characteristics of an effective method of redefining boundary pixels are described with correspond affects on algorithms of outline objects segmentation on bitmap images.

Low-Complexity Acoustic Scene Classification Using Time Frequency Separable Convolution

Replacing 2D-convolution operations by depth-wise separable time and frequency convolutions greatly reduces the number of parameters while maintaining nearly equivalent performances in the context of acoustic scene classification. In our experiments, the models’ sizes can be reduced by 6 to 14 times with similar performances. For a 3-class audio classification, replacing 2D-convolution in a CNN model gives roughly a 2% increase in accuracy. In a 10-class audio classification with multiple recording devices, replacing 2D-convolution in Resnet only reduces around 1.5% of the accuracy.

Advances in Hyperspectral Image Classification with a Bottleneck Attention Mechanism Based on 3D-FCNN Model and Imaging Spectrometer Sensor

Deep learning approaches have significantly enhanced the classification accuracy of hyperspectral images (HSIs). However, the classification process still faces difficulties such as those posed by high data dimensions, large data volumes, and insufficient numbers of labeled samples. To enhance the classification accuracy and reduce the data dimensions and training needed for labeled samples, a 3D fully convolutional neural network (3D-FCNN) model was developed by including a bottleneck attention module. In such a model, the convolutional layer replaces the downsampling layer and the fully connected layer, and 3D full convolution is adopted to replace the commonly used 2D and 1D convolution operations. Thus, the loss of data in the dimensionality reduction process is effectively avoided. The bottleneck attention mechanism is introduced in the FCNN to reduce the redundancy of information and the number of labeled samples. The proposed method was compared to some advanced HSI classification approaches with deep networks, and five common HSI datasets were employed. The experiments showed that our network could achieve considerable classification accuracies by reducing the data dimensionality using a small number of labeled samples, thereby demonstrating its potential merits in the HSI classification process .

A lightweight 3D-2D convolutional neural network for spectral-spatial classification of hyperspectral images

Hyperspectral Image (HSI) is usually composed of hundreds of capturing wavelength bands, which not only increase the size of the HSI rapidly but also impose various obstacles in classifying the objects accurately. Moreover, the traditional machine learning schemes utilize only the spectral features for HSI classification, which, therefore, neglect the spatial features that have a significant impact on the classification improvement. To address the aforementioned issues, in this paper, we propose to employ the principal component analysis (PCA), the baseline feature extraction method, and a thoughtfully designed stacked autoencoder, a deep learning-based feature extraction approach, for reducing the high dimensionality of the HSI and then propose a novel lightweight 3D-2D convolutional neural network (CNN) framework to concurrently exploit both spatial and spectral features from the dimensionality-reduced HSI for classification. In particular, PCA and stacked autoencoder are applied to reduce the high dimensionality of the original HSI and then the proposed 3D-2D CNN provides a combination of 3D and 2D convolution operations to extract the subtle spatial and spectral features for efficient classification. We well-adjust the proposed 3D-2D CNN architecture, and perform extensive experiments on three benchmark HSI datasets and compare our approach with the state-of-the-art classical and deep learning methods. Experimental results illustrate that we have achieved an overall accuracy of 99.73%, 99.90%, and 99.32% on Indian Pines, Pavia University, and Kennedy Space Center datasets, respectively, which outperform the classical machine learning and independent 2D and 3D CNN-based state-of-the-art methods.

Global-Local Spatial-Temporal Residual Correlation Network for Urban Traffic Status Prediction.

The recent proposed Spatial-Temporal Residual Network (ST-ResNet) model is an effective tool to extract both spatial and temporal characteristics and has been successfully applied to urban traffic status prediction. However, the ST-ResNet model only extracts the local spatial characteristics and ignores the very important global spatial characteristics. In this paper, a novel Global-Local Spatial-Temporal Residual Correlation Network (GL-STRCN) model is proposed for urban traffic status prediction to further improve the prediction accuracy of the existing ST-ResNet model. The GL-STRCN model firstly applies Pearson's correlation coefficient method to extract high correlation series. Then, considering both global and local spatial properties, two components consisting of 2D convolution and residual operation are used to capture spatial features. After that, based on Long Short-Term Memory (LSTM) or Gated Recurrent Unit (GRU), a novel long-term temporal feature extraction component is proposed to capture temporal features. Finally, the spatial and temporal features are aggregated together in a weighted way for final prediction. Experiments have also been performed using two datasets from TaxiCD and PEMS-BAY. The results indicated that the proposed model produces a better prediction performance compared with the results based on other baseline solutions, e.g., CNN, ST-ResNet, GL-TCN, and DGLSTNet.

JointE: Jointly utilizing 1D and 2D convolution for knowledge graph embedding

Knowledge graph embedding is a popular method to predict missing links for knowledge graphs by projecting entities and relations into continuous low-dimension embeddings. Some recent embedding models employ translation-based operations to learn the representations of entities and relations with shallow and linear structures, and others leverage neural networks, especially convolution neural networks, to embed the entities and relations with deep and non-linear structures. However, shallow and linear models limit the extraction capacity of the latent knowledge while deep and non-linear models lead to the overabundance of parameters and the loss of surface and explicit knowledge. In this paper, we propose JointE, which utilizes 1D and 2D convolution operations jointly to alleviate these issues effectively. More specifically, we utilize 2D convolution operations to facilitate the interactions between entities and relations, thereby capturing the latent knowledge sufficiently. To reduce the number of parameters significantly, we innovatively construct 2D convolution filters from internal embeddings rather than using external filters which costs plenty of redundant parameters. Furthermore, we appropriately employ 1D convolution filters over input embeddings to extract the surface and explicit knowledge and preserve it by element-wise addition. Experimental evaluation on five benchmark datasets demonstrates that our model outperforms all other state-of-the-art convolution-based models and simultaneously enhances the parameter efficiency.

Visual Analytics: A New Paradigm for Process Monitoring

As a result of recent breakthroughs in computer vision technologies, significant research interest has emerged to encode process data into visual clues and treat process monitoring problems as computer vision tasks. Imaging time-series signals as a feature engineering step forms a new branch of data analytics called ”visual analytics”. In the context of process monitoring, we define visual analytics as the integration of visual representation of the data combined with the use of computer vision tools and analytical reasoning to support decision-making and knowledge extraction from the data. In this work, a novel end-to-end visual analytics pipeline for industrial process fault detection using 1D and 2D convolution operations is proposed. The proposed approach presents a visual representation of data that captures temporal and local features from historical time-series signals. Next, the learned features in a 2D format are visually recognized and classified using 2D convolution operations. Our experimental results demonstrate that this approach achieves better performance on an industrial multivariate dataset compared to other state-of-art signals imaging tools such as Gramian Angular Field (GAF) and Recurrence Plots (RP).

Point cloud semantic scene segmentation based on coordinate convolution

AbstractPoint cloud semantic segmentation, a crucial research area in the 3D computer vision, lies at the core of many vision and robotics applications. Due to the irregular and disordered of the point cloud, however, the application of convolution on point clouds is challenging. In this article, we propose the “coordinate convolution,” which can effectively extract local structural information of the point cloud, to solve the inapplicability of conventional convolution neural network (CNN) structures on the 3D point cloud. The “coordinate convolution” is a projection operation of three planes based on the local coordinate system of each point. Specifically, we project the point cloud on three planes in the local coordinate system with a joint 2D convolution operation to extract its features. Additionally, we leverage a self‐encoding network based on image semantic segmentation U‐Net structure as the overall architecture of the point cloud semantic segmentation algorithm. The results demonstrate that the proposed method exhibited excellent performances for point cloud data sets corresponding to various scenes.

A Real-Time Convolutional Neural Network for Super-Resolution on FPGA With Applications to 4K UHD 60 fps Video Services

In this paper, we present a novel hardware-friendly super-resolution (SR) method based on a convolutional neural network (CNN) and its dedicated hardware (HW) on field programmable gate array (FPGA). Although CNN-based SR methods have shown very promising results for SR, their computational complexities are prohibitive for hardware implementation. To the best of our knowledge, we are the first to implement a real-time CNN-based SR HW that upscales 2K full high-definition video to 4K ultra high-definition (UHD) video at 60 frames per second (fps). In our dedicated CNN-based SR HW, low-resolution input frames are processed line-by-line, and the number of convolutional filter parameters is reduced significantly by incorporating depth-wise separable convolutions with a residual connection. Our CNN-based SR HW incorporates a cascade of 1D convolutions having large receptive fields along horizontal lines while keeping vertical receptive fields minimal, which allows us to save required line memory space in achieving comparable SR performance against full 2D convolution operations. For efficient HW implementation, we use a simple and effective quantization method with little peak signal-to-noise ratio (PSNR) degradation. Also, we propose a compression method to efficiently store intermediate feature map data to reduce the number of line memories used in HW. Our HW implementation on the FPGA generates 4K UHD frames of higher PSNR values at 60 fps and shows better visual quality, compared with conventional CNN-based SR methods that are trained and tested in software.

Protein secondary structure prediction improved by recurrent neural networks integrated with two-dimensional convolutional neural networks.

Protein secondary structure prediction (PSSP) is an important research field in bioinformatics. The representation of protein sequence features could be treated as a matrix, which includes the amino-acid residue (time-step) dimension and the feature vector dimension. Common approaches to predict secondary structures only focus on the amino-acid residue dimension. However, the feature vector dimension may also contain useful information for PSSP. To integrate the information on both dimensions of the matrix, we propose a hybrid deep learning framework, two-dimensional convolutional bidirectional recurrent neural network (2C-BRNN), for improving the accuracy of 8-class secondary structure prediction. The proposed hybrid framework is to extract the discriminative local interactions between amino-acid residues by two-dimensional convolutional neural networks (2DCNNs), and then further capture long-range interactions between amino-acid residues by bidirectional gated recurrent units (BGRUs) or bidirectional long short-term memory (BLSTM). Specifically, our proposed 2C-BRNNs framework consists of four models: 2DConv-BGRUs, 2DCNN-BGRUs, 2DConv-BLSTM and 2DCNN-BLSTM. Among these four models, the 2DConv- models only contain two-dimensional (2D) convolution operations. Moreover, the 2DCNN- models contain 2D convolutional and pooling operations. Experiments are conducted on four public datasets. The experimental results show that our proposed 2DConv-BLSTM model performs significantly better than the benchmark models. Furthermore, the experiments also demonstrate that the proposed models can extract more meaningful features from the matrix of proteins, and the feature vector dimension is also useful for PSSP. The codes and datasets of our proposed methods are available at https://github.com/guoyanb/JBCB2018/ .

2D Convolution Operation with Partial Buffering Implementation on FPGA

2D Convolution Operation with Partial Buffering Implementation on FPGA

Mathematical modeling and application of removal functions during deterministic ion beam figuring of optical surfaces. Part 1: Mathematical modeling.

Ion beam figuring (IBF) is established for the final precision figuring of high-performance optical components, where the figuring accuracy is guaranteed by the stability of the removal function and the solution accuracy of the dwell time. In this deterministic method, the figuring process can be represented by a two-dimensional (2D) convolution operation of a constant removal function and the dwell time. However, we have found that the current 2D convolution operation cannot factually describe the IBF process of curved surfaces, which neglects the influences of the projection distortion and the workpiece geometry on the removal function. Consequently, the current 2D convolution algorithm would influence the solution accuracy for the dwell time and reduce the convergence of the figuring process. In this part, based on the material removal characteristics of IBF, a mathematical model of the removal function is developed theoretically and verified experimentally. Research results show that the removal function during IBF of a curved surface is actually a dynamic function in the 2D convolution algorithm. The mathematical modeling of the dynamic removal function provides theoretical foundations for our proposed new algorithm in the next part, and final verification experiments indicate that this algorithm can effectively improve the accuracy of the dwell time solution for the IBF of curved surfaces.

Mathematical modeling and application of removal functions during deterministic ion beam figuring of optical surfaces. Part 2: application.

Ion beam figuring (IBF) is established for the final precision figuring of optical components. In this deterministic method, the figuring process is represented by a two-dimensional (2D) convolution operation of a constant removal function and the dwell time, where the figuring precision is guaranteed by the stability of the removal function as well as the solution accuracy of the dwell time. However, the current 2D convolution equation cannot factually reflect the IBF process of curved surfaces, which neglects the influence of the projection distortion and the workpiece geometry. Consequently, the current convolution algorithm for the IBF process would influence the solution accuracy for the dwell time and reduce the convergence of the figuring process. In this part, we propose an improved algorithm based on the mathematical modeling of the dynamic removal function in Part A, which provides a more accurate dwell time for IBF of a curved surface. Additionally, simulation analysis and figuring experiments are carried out to verify the feasibility of our proposed algorithm. The final experimental results indicate that the figuring precision and efficiency can be simultaneously improved by this method.

Comparison between Frame-Constrained Fix-Pixel-Value and Frame-Free Spiking-Dynamic-Pixel ConvNets for Visual Processing

Most scene segmentation and categorization architectures for the extraction of features in images and patches make exhaustive use of 2D convolution operations for template matching, template search, and denoising. Convolutional Neural Networks (ConvNets) are one example of such architectures that can implement general-purpose bio-inspired vision systems. In standard digital computers 2D convolutions are usually expensive in terms of resource consumption and impose severe limitations for efficient real-time applications. Nevertheless, neuro-cortex inspired solutions, like dedicated Frame-Based or Frame-Free Spiking ConvNet Convolution Processors, are advancing real-time visual processing. These two approaches share the neural inspiration, but each of them solves the problem in different ways. Frame-Based ConvNets process frame by frame video information in a very robust and fast way that requires to use and share the available hardware resources (such as: multipliers, adders). Hardware resources are fixed- and time-multiplexed by fetching data in and out. Thus memory bandwidth and size is important for good performance. On the other hand, spike-based convolution processors are a frame-free alternative that is able to perform convolution of a spike-based source of visual information with very low latency, which makes ideal for very high-speed applications. However, hardware resources need to be available all the time and cannot be time-multiplexed. Thus, hardware should be modular, reconfigurable, and expansible. Hardware implementations in both VLSI custom integrated circuits (digital and analog) and FPGA have been already used to demonstrate the performance of these systems. In this paper we present a comparison study of these two neuro-inspired solutions. A brief description of both systems is presented and also discussions about their differences, pros and cons.