3D Convolution Research Articles

Deep Learning Based Underwater Acoustic Target Recognition: Introduce a Recent Temporal 2D Modeling Method.

In recent years, the application of deep learning models for underwater target recognition has become a popular trend. Most of these are pure 1D models used for processing time-domain signals or pure 2D models used for processing time-frequency spectra. In this paper, a recent temporal 2D modeling method is introduced into the construction of ship radiation noise classification models, combining 1D and 2D. This method is based on the periodic characteristics of time-domain signals, shaping them into 2D signals and discovering long-term correlations between sampling points through 2D convolution to compensate for the limitations of 1D convolution. Integrating this method with the current state-of-the-art model structure and using samples from the Deepship database for network training and testing, it was found that this method could further improve the accuracy (0.9%) and reduce the parameter count (30%), providing a new option for model construction and optimization. Meanwhile, the effectiveness of training models using time-domain signals or time-frequency representations has been compared, finding that the model based on time-domain signals is more sensitive and has a smaller storage footprint (reduced to 30%), whereas the model based on time-frequency representation can achieve higher accuracy (1-2%).

Segmentation method of magnetic resonance imaging brain tumor images based on improved UNet network.

Glioma is a primary malignant craniocerebral tumor commonly found in the central nervous system. According to research, preoperative diagnosis of glioma and a full understanding of its imaging features are very significant. Still, the traditional segmentation methods of image dispensation and machine wisdom are not acceptable in glioma segmentation. This analysis explores the potential of magnetic resonance imaging (MRI) brain tumor images as an effective segmentation method of glioma. This study used 200 MRI images from the affiliated hospital and applied the 2-dimensional residual block UNet (2DResUNet). Features were extracted from input images using a 2×2 kernel size (64-kernel) 1-step 2D convolution (Conv) layer. The 2DDenseUNet model implemented in this study incorporates a ResBlock mechanism within the UNet architecture, as well as a Gaussian noise layer for data augmentation at the input stage, and a pooling layer for replacing the conventional 2D convolutional layers. Finally, the performance of the proposed protocol and its effective measures in glioma segmentation were verified. The outcomes of the 5-fold cross-validation evaluation show that the proposed 2DResUNet and 2DDenseUNet structure has a high sensitivity despite the slightly lower evaluation result on the Dice score. At the same time, compared with other models used in the experiment, the DM-DA-UNet model proposed in this paper was significantly improved in various indicators, increasing the reliability of the model and providing a reference and basis for the accurate formulation of clinical treatment strategies. The method used in this study showed stronger feature extraction ability than the UNet model. In addition, our findings demonstrated that using generalized die harm and prejudiced cross entropy as loss functions in the training process effectively alleviated the class imbalance of glioma data and effectively segmented glioma. The method based on the improved UNet network has obvious advantages in the MRI brain tumor portrait segmentation procedure. The result showed that we developed a 2D residual block UNet, which can improve the incorporation of glioma segmentation into the clinical process.

3D-IncNet: Head and Neck (H&N) Primary Tumors Segmentation and Survival Prediction.

Cancer begins when healthy cells change and grow out of control, forming a mass called a tumor. Head and neck (H&N) cancers usually develop in or around the head and neck, including the mouth (oral cavity), nose and sinuses, throat (pharynx), and voice box (larynx). 4% of all cancers are H&N cancers with a very low survival rate (a five-year survival rate of 64.7%). FDG-PET/CT imaging is often used for early diagnosis and staging of H&N tumors, thus improving these patients' survival rates. This work presents a novel 3D-Inception-Residual aided with 3D depth-wise convolution and squeeze and excitation block. We introduce a 3D depth-wise convolution-inception encoder consisting of an additional 3D squeeze and excitation block and a 3D depth-wise convolution-based residual learning decoder (3D-IncNet), which not only helps to recalibrate the channel-wise features but adaptively through explicit inter-dependencies modeling but also integrate the coarse and fine features resulting in accurate tumor segmentation. We further demonstrate the effectiveness of inception-residual encoder-decoder architecture in achieving better dice scores and the impact of depth-wise convolution in lowering the computational cost. We applied random forest for survival prediction on deep, clinical, and radiomics features. Experiments are conducted on the benchmark HECKTOR21 challenge, which showed significantly better performance by surpassing the state-of-the-artwork and achieved 0.836 and 0.811 concordance index and dice scores, respectively. We made the model and code publicly available.

Riesz Networks: Scale-Invariant Neural Networks in a Single Forward Pass

AbstractScale invariance of an algorithm refers to its ability to treat objects equally independently of their size. For neural networks, scale invariance is typically achieved by data augmentation. However, when presented with a scale far outside the range covered by the training set, neural networks may fail to generalize. Here, we introduce the Riesz network, a novel scale- invariant neural network. Instead of standard 2d or 3d convolutions for combining spatial information, the Riesz network is based on the Riesz transform which is a scale-equivariant operation. As a consequence, this network naturally generalizes to unseen or even arbitrary scales in a single forward pass. As an application example, we consider detecting and segmenting cracks in tomographic images of concrete. In this context, ‘scale’ refers to the crack thickness which may vary strongly even within the same sample. To prove its scale invariance, the Riesz network is trained on one fixed crack width. We then validate its performance in segmenting simulated and real tomographic images featuring a wide range of crack widths. An additional experiment is carried out on the MNIST Large Scale data set.

Real-time and accurate deep learning-based multi-organ nucleus segmentation in histology images

Automated nucleus segmentation is considered the gold standard for diagnosing some severe diseases. Accurate instance segmentation of nuclei is still very challenging because of the large number of clustered nuclei, and the different appearance of nuclei for different tissue types. In this paper, a neural network is proposed for fast and accurate instance segmentation of nuclei in histopathology images. The network is inspired by the Unet and residual nets. The main contribution of the proposed model is enhancing the classification accuracy of nuclear boundaries by moderately preserving the spatial features by relatively d the size of feature maps. Then, a proposed 2D convolution layer is used instead of the conventional 3D convolution layer, the core of CNN-based architectures, where the feature maps are first compacted before being convolved by 2D kernel filters. This significantly reduces the processing time and avoids the out of memory problem of the GPU. Also, more features are extracted when getting deeper into the network without degrading the spatial features dramatically. Hence, the number of layers, required to compensate the loss of spatial features, is reduced that also reduces the processing time. The proposed approach is applied to two multi-organ datasets and evaluated by the Aggregated Jaccard Index (AJI), F1-score and the number of frames per second. Also, the formula of AJI is modified to reflect the object- and pixel-level errors more accurately. The proposed model is compared to some state-of-the-art architectures, and it shows better performance in terms of the segmentation speed and accuracy.

A Hyperspectral Change Detection (HCD-Net) Framework Based on Double Stream Convolutional Neural Networks and an Attention Module

Human activities and natural phenomena continually transform the Earth’s surface, presenting ongoing challenges to the environment. Therefore, the accurate and timely monitoring and prediction of these alterations are essential for devising effective solutions and mitigating environmental impacts in advance. This study introduces a novel framework, called HCD-Net, for detecting changes using bi-temporal hyperspectral images. HCD-Net is built upon a dual-stream deep feature extraction process, complemented by an attention mechanism. The first stream employs 3D convolution layers and 3D Squeeze-and-Excitation (SE) blocks to extract deep features, while the second stream utilizes 2D convolution and 2D SE blocks for the same purpose. The deep features from both streams are then concatenated and processed through dense layers for decision-making. The performance of HCD-Net is evaluated against existing state-of-the-art change detection methods. For this purpose, the bi-temporal Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) hyperspectral dataset was utilized to assess the change detection performance. The findings indicate that HCD-Net achieves superior accuracy and the lowest false alarm rate among the compared methods, with an overall classification accuracy exceeding 96%, and a kappa coefficient greater than 0.9.

Atom-ProteinQA: Atom-level protein model quality assessment through fine-grained joint learning

MotivationProtein model quality assessment (ProteinQA) is a fundamental task that is essential for biologically relevant applications, i.e., protein structure refinement, protein design, etc. Previous works aimed to conduct ProteinQA only on the global structure or per-residue level, ignoring potentially usable and precise cues from a fine-grained per-atom perspective. In this study, we propose an atom-level ProteinQA model, named Atom-ProteinQA, in which two innovative modules are designed to extract geometric and topological atom-level relationships respectively. Specifically, on the one hand, a geometric perception module exploits 3D sparse convolution to capture the geometric features of the input protein, generating fine-grained atom-level predictions. On the other hand, natural chemical bonds are utilized to construct an atom-level graph, then message passing from a topological perception module is applied to output residue-level predictions in parallel. Eventually, through a cross-model aggregation module, features from different modules mutually interact, enhancing performance on both the atom and residue levels. ResultsExtensive experiments show that our proposed Atom-ProteinQA outperforms previous methods by a large margin, regardless of residue-level or atom-level assessment. Concretely, we achieved state-of-the-art performance on CATH-2084, Decoy-8000, public benchmarks CASP13 & CASP14, and the CAMEO. AvailabilityThe repository of this project is released on: https://github.com/luyfcandy/Atom_ProteinQA.

Proformer: a hybrid macaron transformer model predicts expression values from promoter sequences

The breakthrough high-throughput measurement of the cis-regulatory activity of millions of randomly generated promoters provides an unprecedented opportunity to systematically decode the cis-regulatory logic that determines the expression values. We developed an end-to-end transformer encoder architecture named Proformer to predict the expression values from DNA sequences. Proformer used a Macaron-like Transformer encoder architecture, where two half-step feed forward (FFN) layers were placed at the beginning and the end of each encoder block, and a separable 1D convolution layer was inserted after the first FFN layer and in front of the multi-head attention layer. The sliding k-mers from one-hot encoded sequences were mapped onto a continuous embedding, combined with the learned positional embedding and strand embedding (forward strand vs. reverse complemented strand) as the sequence input. Moreover, Proformer introduced multiple expression heads with mask filling to prevent the transformer models from collapsing when training on relatively small amount of data. We empirically determined that this design had significantly better performance than the conventional design such as using the global pooling layer as the output layer for the regression task. These analyses support the notion that Proformer provides a novel method of learning and enhances our understanding of how cis-regulatory sequences determine the expression values.

Hybrid 2D–3D convolution and pre-activated residual networks for hyperspectral image classification

Hybrid 2D–3D convolution and pre-activated residual networks for hyperspectral image classification

STGWN: Enhanced spatiotemporal wave forecasting using multiscale features

This paper addresses the challenge of low accuracy in wave height prediction due to inadequate consideration of irregular topological structures and spatio-temporal dependencies of wave measurement points. We propose a novel spatio-temporal wave prediction method, named the Spatio-Temporal Graph Wave Network (STGWN). This model employs graph convolution instead of conventional 2D convolution, enabling spatial domain modeling of wave fields on a graph structure. Simultaneously, it utilizes LSTM for temporal modeling, effectively enhancing the representation of complex dynamic spatio-temporal correlations among wave nodes. Additionally, a dynamic wave position-aware mechanism is constructed to better identify feature variations among different wave nodes. Furthermore, a multi-scale feature fusion mechanism is designed within the model to prevent the risk of crucial temporal feature disappearance during the extraction of spatial dimension features. The proposed spatio-temporal method for Significant Wave Height (SWH) prediction is validated on a wave dataset comprising four different node distribution characteristics within the study area. Results demonstrate that the STGWN model outperforms comparative models in terms of prediction accuracy and stability across multiple evaluation metrics in the four experimental setups.

AST3DRNet: Attention-Based Spatio-Temporal 3D Residual Neural Networks for Traffic Congestion Prediction.

Traffic congestion prediction has become an indispensable component of an intelligent transport system. However, one limitation of the existing methods is that they treat the effects of spatio-temporal correlations on traffic prediction as invariable during modeling spatio-temporal features, which results in inadequate modeling. In this paper, we propose an attention-based spatio-temporal 3D residual neural network, named AST3DRNet, to directly forecast the congestion levels of road networks in a city. AST3DRNet combines a 3D residual network and a self-attention mechanism together to efficiently model the spatial and temporal information of traffic congestion data. Specifically, by stacking 3D residual units and 3D convolution, we proposed a 3D convolution module that can simultaneously capture various spatio-temporal correlations. Furthermore, a novel spatio-temporal attention module is proposed to explicitly model the different contributions of spatio-temporal correlations in both spatial and temporal dimensions through the self-attention mechanism. Extensive experiments are conducted on a real-world traffic congestion dataset in Kunming, and the results demonstrate that AST3DRNet outperforms the baselines in short-term (5/10/15 min) traffic congestion predictions with an average accuracy improvement of 59.05%, 64.69%, and 48.22%, respectively.

Improving 3D Object Detection with Context-Aware and Dimensional Interaction Attention

Recently, 3D object detection technology based on point clouds has developed rapidly. However, too few points of distant and occluded objects are scanned by the sensor, and thus these objects suffer from too insufficient features to be detected. This case damages the detection accuracy. Therefore, we constitute a novel 3D object detection with Context-aware and dimensional Interaction Attention Network (CIANet) to explore vital geometric cues for enriching the feature representation of the object, thus boosting the overall detection performance. Specifically, in the first stage, we employ the 3D sparse convolution to extract voxel features, and then construct a Channel-Spatial Hybrid Attention (CSHA) module and a Contextual Self-Attention (CSA) module to enhance voxel features for generating proposals. The CSHA module aims to enhance the key information of the channel and spatial domains of 2D Bird’s Eye View (BEV) features, and the CSA module is applied to supplement contextual information to the enhanced BEV features, thus generating accurate proposals. In the second stage, we construct a Dimensional Interaction Attention (DIA) module to refine Region of Interest (RoI) features within the proposals. It enhances the interactions among the channel and spatial dimensions of RoI features to learn accurate boundaries of objects for proposal refinement. Extensive experiments on the KITTI and Waymo benchmarks show the superior detection performance of CIANet compared to recent methods, especially for objects such as pedestrians and cyclists.

3DUV-NetR+: A 3D hybrid semantic architecture using transformers for brain tumor segmentation with MultiModal MR images

Brain tumor segmentation plays a substantial role in Medical Image Analysis (MIS). In this regard, automatic segmentation methods facilitate precise and efficient segmentation, significantly contributing to diagnosis and treatment planning in medical applications. Recently, several Deep Learning-based architectures have been proposed to revolutionize the MIS field. Particularly, the combination of Convolution Neural Networks (CNNs) and Transformers has greatly enhanced and developed segmentation results. Moreover, the Attention mechanism in Transformers allows the modeling of long-range contextual features extracted from CNNs' encoder part. This paper proposes a hybrid advanced 3D model for brain tumor segmentation using multi-modal magnetic resonance images. The model benefits from the features extracted from the encoder of 3DU-Net and V-Net architectures at each depth. Then, a concatenation between these features and their fusion is carried out at each decoder depth to build new significant features followed by a 3D convolution layer and Transformers block for more contextual information. In addition, a final convolution block is applied to get the segmented tumor. To this end, the model is evaluated on the BraTS 2020 dataset to segment different sub-regions of brain tumors. The obtained results demonstrate the effectiveness of the proposed model in terms of dice similarity coefficient (DSC) and Hausdorff Distance (HD). For DSC, 91.95% and 82.80% and 81.70% for Whole Tumor(WT), Tumor Core (TC), and Enhancing Tumor(ET), respectively are archived, while for HD, 4.9 mm, 6.0 mm and 3.8 mm for WT, TC and ET are accomplished.

Volumetric Imitation Generative Adversarial Networks for Anatomical Human Body Modeling.

Volumetric representation is a technique used to express 3D objects in various fields, such as medical applications. On the other hand, tomography images for reconstructing volumetric data have limited utilization because they contain personal information. Existing GAN-based medical image generation techniques can produce virtual tomographic images for volume reconstruction while preserving the patient's privacy. Nevertheless, these images often do not consider vertical correlations between the adjacent slices, leading to erroneous results in 3D reconstruction. Furthermore, while volume generation techniques have been introduced, they often focus on surface modeling, making it challenging to represent the internal anatomical features accurately. This paper proposes volumetric imitation GAN (VI-GAN), which imitates a human anatomical model to generate volumetric data. The primary goal of this model is to capture the attributes and 3D structure, including the external shape, internal slices, and the relationship between the vertical slices of the human anatomical model. The proposed network consists of a generator for feature extraction and up-sampling based on a 3D U-Net and ResNet structure and a 3D-convolution-based LFFB (local feature fusion block). In addition, a discriminator utilizes 3D convolution to evaluate the authenticity of the generated volume compared to the ground truth. VI-GAN also devises reconstruction loss, including feature and similarity losses, to converge the generated volumetric data into a human anatomical model. In this experiment, the CT data of 234 people were used to assess the reliability of the results. When using volume evaluation metrics to measure similarity, VI-GAN generated a volume that realistically represented the human anatomical model compared to existing volume generation methods.

A novel DenseNet with warm restarts for gas recognition in complex airflow environments

The diffusion mechanisms in natural environments often result in irregular and turbulent flow patterns in electronic nose field. This leads to random responses of electronic nose when exposed to the same gas ambiance, and thus brings serious challenges in gas identification. Recent studies have shown that the introducing of skip connections can enhance the capability of neural networks to extract features from gas data. In response to this finding, a novel One-dimensional DenseNet with Warm Restarts (1D-DNR) comprising skip connections is presented in this study. The network combines shallow and deep features using DenseNet and extracts the signal relevant to the unique characteristics of gas data through a 1D convolution. Furthermore, the classification accuracy and the stability of the proposed network are enhanced via the cosine annealing with warm restarts method. Further, zero-offset baseline compensation and Discrete Wavelet Transform (DWT) were employed to correct long-term drift and short-term drift effects. The results revealed that it is superior to the other models when dealing with the data of irregular and turbulent flow patterns of gases, with a classification accuracy of 99.83%.

Unsupervised 3D reconstruction method based on multi-view propagation

In this paper, an end-to-end deep learning framework for reconstructing 3D models by computing depth maps from multiple views is proposed. An unsupervised 3D reconstruction method based on multi-view propagation is introduced, which addresses the issues of large GPU memory consumption caused by most current research methods using 3D convolution for 3D cost volume regularization and regression to obtain the initial depth map, as well as the difficulty in obtaining true depth values in supervised methods due to device limitations. The method is inspired by the Patchmatch algorithm, and the depth is divided into n layers within the depth range to obtain depth hypotheses through multi-view propagation. What's more, a multi-metric loss function is constructed based on luminosity consistency, structural similarity, and depth smoothness between multiple views to serve as a supervisory signal for learning depth predictions in the network. The experimental results show our proposed method has a very competitive performance and generalization on the DTU, Tanks & Temples and our self-made dataset; Specifically, it is at least 1.7 times faster and requires more than 75% less memory than the method that utilizes 3D cost volume regularization.

Unified Spatio-Temporal Attention Models for Advanced Human Action Recognition & Detection

Abstract: This research paper introduces innovative approaches to enhance human action analytics in computer vision through the development of spatial and temporal attention models. The first model is based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units, aiming to extract discriminative spatial and temporal features from skeleton data for improved human action recognition and detection. The model selectively focuses on discriminative joints within each frame, utilizing a regularized cross-entropy loss for effective training. Additionally, a joint training strategy is proposed to optimize the learning process. The second model, a spatio-temporal attention (STA) network, is designed to address the limitations of existing 3D Convolutional Neural Networks (3D CNNs) in treating all video frames equally. The STA network characterizes beneficial information at both the frame and channel levels, leveraging differences in spatial and temporal dimensions to enhance the learning capability of 3D convolutions. The proposed STA method can be seamlessly integrated into state-of-the-art 3D CNN architectures for video action detection and recognition. Evaluation of diverse datasets, including SBU Kinect Interaction, NTU RGB+D, PKU-MMD, UCF-101, HMDB-51, and THUMOS 2014, demonstrates the effectiveness of both proposed models in achieving state-of-the-art performance for action recognition and detection tasks. The spatial and temporal attention models contribute significantly to capturing discriminative features, showcasing their potential in advancing the field of human action analytics in computer vision.

Efficient FPGA Implementation of Convolutional Neural Networks and Long Short-Term Memory for Radar Emitter Signal Recognition

In recent years, radar emitter signal recognition has enjoyed a wide range of applications in electronic support measure systems and communication security. More and more deep learning algorithms have been used to improve the recognition accuracy of radar emitter signals. However, complex deep learning algorithms and data preprocessing operations have a huge demand for computing power, which cannot meet the requirements of low power consumption and high real-time processing scenarios. Therefore, many research works have remained in the experimental stage and cannot be actually implemented. To tackle this problem, this paper proposes a resource reuse computing acceleration platform based on field programmable gate arrays (FPGA), and implements a one-dimensional (1D) convolutional neural network (CNN) and long short-term memory (LSTM) neural network (NN) model for radar emitter signal recognition, directly targeting the intermediate frequency (IF) data of radar emitter signal for classification and recognition. The implementation of the 1D-CNN-LSTM neural network on FPGA is realized by multiplexing the same systolic array to accomplish the parallel acceleration of 1D convolution and matrix vector multiplication operations. We implemented our network on Xilinx XCKU040 to evaluate the effectiveness of our proposed solution. Our experiments show that the system can achieve 7.34 giga operations per second (GOPS) data throughput with only 5.022 W power consumption when the radar emitter signal recognition rate is 96.53%, which greatly improves the energy efficiency ratio and real-time performance of the radar emitter recognition system.

Deep video inpainting detection and localization based on ConvNeXt dual-stream network

Currently, deep learning-based video inpainting algorithms can fill in a specified video region with visually plausible content, usually leaving imperceptible traces. Since deep video inpainting methods can be used to maliciously manipulate video content, there is an urgent need for an effective method to detect and localize deep video inpainting. In this paper, we propose a dual-stream video inpainting detection network, which includes a ConvNeXt dual-stream encoder and a multi-scale feature cross-fusion decoder. To further explore the spatial and temporal traces left by deep inpainting, we extract motion residuals and enhance them using 3D convolution and SRM filtering. Furthermore, we extract filtered residuals using LoG and Laplacian filtering. These residuals are then entered into ConvNeXt, thereby learning discriminative inpainting features. To enhance detection accuracy, we design a top-down pyramid decoder that aims at deep fusion of multi-dimensional multi-scale features to fully exploit the information of different dimensions and levels in detail. We created two datasets containing state-of-the-art video inpainting algorithms and conducted various experiments to evaluate our approach. The experimental results demonstrate that our approach outperforms existing methods and attains a competitive performance despite encountering unseen inpainting algorithms.

Multimodal diagnosis model of Alzheimer’s disease based on improved Transformer

PurposeRecent technological advancements in data acquisition tools allowed neuroscientists to acquire different modality data to diagnosis Alzheimer’s disease (AD). However, how to fuse these enormous amount different modality data to improve recognizing rate and find significance brain regions is still challenging.MethodsThe algorithm used multimodal medical images [structural magnetic resonance imaging (sMRI) and positron emission tomography (PET)] as experimental data. Deep feature representations of sMRI and PET images are extracted by 3D convolution neural network (3DCNN). An improved Transformer is then used to progressively learn global correlation information among features. Finally, the information from different modalities is fused for identification. A model-based visualization method is used to explain the decisions of the model and identify brain regions related to AD.ResultsThe model attained a noteworthy classification accuracy of 98.1% for Alzheimer’s disease (AD) using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset. Upon examining the visualization results, distinct brain regions associated with AD diagnosis were observed across different image modalities. Notably, the left parahippocampal region emerged consistently as a prominent and significant brain area.ConclusionsA large number of comparative experiments have been carried out for the model, and the experimental results verify the reliability of the model. In addition, the model adopts a visualization analysis method based on the characteristics of the model, which improves the interpretability of the model. Some disease-related brain regions were found in the visualization results, which provides reliable information for AD clinical research.