3D CNNs Research Articles

Ensemble Vision Transformer for Dementia Diagnosis.

In recent years, deep learning has gained momentum in computer-aided Alzheimer's Disease (AD) diagnosis. This study introduces a novel approach, Monte Carlo Ensemble Vision Transformer (MC-ViT), which develops an ensemble approach with Vision transformer (ViT). Instead of using traditional ensemble methods that deploy multiple learners, our approach employs a single vision transformer learner. By harnessing Monte Carlo sampling, this method produces a broad spectrum of classification decisions, enhancing the MC-ViT performance. This novel technique adeptly overcomes the limitation of 3D patch convolutional neural networks that only characterize partial of the whole brain anatomy, paving the way for a neural network adept at discerning 3D inter-feature correlations. Evaluations using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset with 7199 scans and Open Access Series of Imaging Studies-3 (OASIS-3) with 1992 scans showcased its performance. With minimal preprocessing, our approach achieved an impressive 90% accuracy in AD classification, surpassing both 2D-slice CNNs and 3D CNNs.

Enhancing Brain Tumor Classification with a Novel Three-Dimensional Convolutional Neural Network (3D-CNN) Fusion Model

Three-dimensional convolutional neural networks (3D CNNs) have been widely applied to analyze brain tumour images (BT) to understand the disease's progress better. It is well-known that training 3D-CNN is computationally expensive and has the potential of overfitting due to the small sample size available in the medical imaging field. Here, we proposed a novel 2D-3D approach by converting a 2D brain image to a 3D fused image using a gradient of the image Learnable Weighted. By the 2D-to-3D conversion, the proposed model can easily forward the fused 3D image through a pre-trained 3D model while achieving better performance over different 3D baselines. We used VGG16 for feature extraction in the implementation as it outperformed other 3D CNN backbones. We further showed that the weights of the slices are location-dependent, and the model performance relies on the 3D-to-2D fusion view, with the best outcomes from the coronal view. With the new approach, we increased the accuracy to 0.88, compared with conventional 3D CNNs, for classifying brain tumour images. The novel 2D-3D model may have profound implications for future timely BT diagnosis in clinical settings.

Multi-resolution short-time Fourier transform providing deep features for 3D CNN to classify rolling bearing fault vibration signals

The time-frequency domain features of vibration signals provide valuable information for deep learning-based rolling bearing fault diagnosis methods, where fault signal classification aiding in the identification of nominal fault types during diagnosis. The Short-Time Fourier Transform (STFT) is a widely used time-frequency transformation method, and its window length is the key parameter that determines the trade-off between time and frequency resolution. The primary motivation of this study is to address the limitation in traditional STFT-based 2D CNN methods: the inability to adapt the window length to different types of signals. To achieve accurate classification of bearing fault types, this study proposes a method based on three-dimensional convolutional neural networks (3D CNNs) to deeply explore the time-frequency domain information of one-dimensional vibration signals from faulty bearings. This method first applies STFT with multiple window sizes to perform multi-resolution time-frequency transformations on the time-domain vibration signals, yielding three-dimensional data. Subsequently, a classifier is trained based on the proposed 3D CNN. Experimental results on public datasets show that, without any sophisticated techniques, the proposed method achieves an average classification accuracy of 99.2% for six types of bearing faults using a relatively simple CNN structure. Compared to 1D CNN and 2D CNN methods that use fixed window sizes for STFT, the proposed method significantly enhances classification performance. Furthermore, it demonstrates robust classification results even on small-scaled bearing datasets.

Bidirectional Hybrid LSTM Based Recurrent Neural Network for Multi-View Stereo.

Recently, deep learning based multi-view stereo (MVS) networks have demonstrated their excellent performance on various benchmarks. In this paper, we present an effective and efficient recurrent neural network (RNN) for accurate and complete dense point cloud reconstruction. Instead of regularizing the cost volume via conventional 3D CNN or unidirectional RNN like previous attempts, we adopt a bidirectional hybrid Long Short-Term Memory (LSTM) based structure for cost volume regularization. The proposed bidirectional recurrent regularization is able to perceive full-space context information comparable to 3D CNNs while saving runtime memory. For post-processing, we introduce a visibility based approach for depth map refinement to obtain more accurate dense point clouds. Extensive experiments on DTU, Tanks and Temples and ETH3D datasets demonstrate that our method outperforms previous state-of-the-art MVS methods and exhibits high memory efficiency at runtime.

AI-based visual speech recognition towards realistic avatars and lip-reading applications in the metaverse

The metaverse, a virtually shared digital world where individuals interact, create, and explore, has witnessed rapid evolution and widespread adoption. Communication between avatars is crucial to their actions in the metaverse. Advances in natural language processing have allowed for significant progress in producing spoken conversations. Within this digital landscape, the integration of Visual Speech Recognition (VSR) powered by deep learning emerges as a transformative application. This research delves into the concept and implications of VSR in the metaverse. This study focuses on developing realistic avatars and a lip-reading application within the metaverse, utilizing Artificial Intelligence (AI) techniques for visual speech recognition. Visual Speech Recognition in the metaverse refers to using deep learning techniques to comprehend and respond to spoken language, relying on the visual cues provided by users' avatars. This multidisciplinary approach combines computer vision and natural language processing, enabling avatars to understand spoken words by analyzing the movements of their lips and facial expressions. Key components encompass the collection of extensive video datasets, the employment of 3D Convolutional Neural Networks (3D CNNs) combined with ShuffleNet and Densely Connected Temporal Convolutional Neural Networks (DC-TCN) called (CFS-DCTCN) to model visual and temporal features, and the integration of contextual understanding mechanisms. The two datasets Wild (LRW) dataset and the GRID Corpus datasets are utilized to validate the proposed model. As the metaverse continues its prominence, integrating Visual Speech Recognition through deep learning represents a pivotal step towards forging immersive and dynamic virtual worlds where communication transcends physical boundaries. This paper contributes to the foundation of technology-driven metaverse development and fosters a future where digital interactions mirror the complexities of human communication. The proposed model achieves 99.5 % on LRW and 98.8 % on the GRID dataset.

Stereoscopic video deblurring transformer

Stereoscopic cameras, such as those in mobile phones and various recent intelligent systems, are becoming increasingly common. Multiple variables can impact the stereo video quality, e.g., blur distortion due to camera/object movement. Monocular image/video deblurring is a mature research field, while there is limited research on stereoscopic content deblurring. This paper introduces a new Transformer-based stereo video deblurring framework with two crucial new parts: a self-attention layer and a feed-forward layer that realizes and aligns the correlation among various video frames. The traditional fully connected (FC) self-attention layer fails to utilize data locality effectively, as it depends on linear layers for calculating attention maps The Vision Transformer, on the other hand, also has this limitation, as it takes image patches as inputs to model global spatial information. 3D convolutional neural networks (3D CNNs) process successive frames to correct motion blur in the stereo video. Besides, our method uses other stereo-viewpoint information to assist deblurring. The parallax attention module (PAM) is significantly improved to combine the stereo and cross-view information for more deblurring. An extensive ablation study validates that our method efficiently deblurs the stereo videos based on the experiments on two publicly available stereo video datasets. Experimental results of our approach demonstrate state-of-the-art performance compared to the image and video deblurring techniques by a large margin.

Automatic segmentation and classification of frontal sinuses for sex determination from CBCT scans using a two-stage anatomy-guided attention network

Sex determination is essential for identifying unidentified individuals, particularly in forensic contexts. Traditional methods for sex determination involve manual measurements of skeletal features on CBCT scans. However, these manual measurements are labor-intensive, time-consuming, and error-prone. The purpose of this study was to automatically and accurately determine sex on a CBCT scan using a two-stage anatomy-guided attention network (SDetNet). SDetNet consisted of a 2D frontal sinus segmentation network (FSNet) and a 3D anatomy-guided attention network (SDNet). FSNet segmented frontal sinus regions in the CBCT images and extracted regions of interest (ROIs) near them. Then, the ROIs were fed into SDNet to predict sex accurately. To improve sex determination performance, we proposed multi-channel inputs (MSIs) and an anatomy-guided attention module (AGAM), which encouraged SDetNet to learn differences in the anatomical context of the frontal sinus between males and females. SDetNet showed superior sex determination performance in the area under the receiver operating characteristic curve, accuracy, Brier score, and specificity compared with the other 3D CNNs. Moreover, the results of ablation studies showed a notable improvement in sex determination with the embedding of both MSI and AGAM. Consequently, SDetNet demonstrated automatic and accurate sex determination by learning the anatomical context information of the frontal sinus on CBCT scans.

An end-to-end multi-scale airway segmentation framework based on pulmonary CT image

Objective. Automatic and accurate airway segmentation is necessary for lung disease diagnosis. The complex tree-like structures leads to gaps in the different generations of the airway tree, and thus airway segmentation is also considered to be a multi-scale problem. In recent years, convolutional neural networks have facilitated the development of medical image segmentation. In particular, 2D CNNs and 3D CNNs can extract different scale features. Hence, we propose a two-stage and 2D + 3D framework for multi-scale airway tree segmentation. Approach. In stage 1, we use a 2D full airway SegNet(2D FA-SegNet) to segment the complete airway tree. Multi-scale atros spatial pyramid and Atros Residual Skip connection modules are inserted to extract different scales feature. We designed a hard sample selection strategy to increase the proportion of intrapulmonary airway samples in stage 2. 3D airway RefineNet (3D ARNet) as stage 2 takes the results of stage 1 as a priori information. Spatial information extracted by 3D convolutional kernel compensates for the loss of in 2D FA-SegNet. Furthermore, we added false positive losses and false negative losses to improve the segmentation performance of airway branches within the lungs. Main results. We performed data enhancement on the publicly available dataset of ISICDM 2020 Challenge 3, and on which evaluated our method. Comprehensive experiments show that the proposed method has the highest dice similarity coefficient (DSC) of 0.931, and IoU of 0.871 for the whole airway tree and DSC of 0.699, and IoU of 0.543 for the intrapulmonary bronchi tree. In addition, 3D ARNet proposed in this paper cascaded with other state-of-the-art methods to increase detected tree length rate by up to 46.33% and detected tree branch rate by up to 42.97%. Significance. The quantitative and qualitative evaluation results show that our proposed method performs well in segmenting the airway at different scales.

G-TRACE: Grouped temporal recalibration for video object segmentation

In Semi-supervised Video Object Segmentation (SVOS), there is a critical emphasis on enhancing the memory and readout mechanisms for frame matching, especially in relation to temporal dynamics. Current methods predominantly use 2D CNNs for encoding video frames, which unfortunately neglects the crucial aspect of addressing temporal variations in individual frames and their associated masks during the encoding process. One potential solution would be to implement temporal models such as 3D CNNs instead of 2D CNNs, but this significantly increases computational requirements, making it impractical for real-world SVOS applications. In this paper, we introduce the Grouped Temporal Recalibration with Attention for Convolutional Encoders (G-TRACE), a novel plug-and-play module that is compatible with various existing SVOS frameworks. G-TRACE uses hierarchical memory-centric attention and integrates effortlessly with 2D CNNs, offering a novel approach to temporal modeling that operates orthogonally to traditional frame matching methods. Extensive evaluations on four widely-used benchmarks demonstrate that our method consistently delivers significant performance improvements over various baseline models.

Advancements in Motion Detection Within Video Streams Through the Integration of Optical Flow Estimation and 3D-Convolutional Neural Network Architectures

Motion detection in video streams is important for many applications, such as autonomous navigation, human-computer interaction, and spying. Conventional techniques, which depend on backdrop removal or frame differencing, frequently break down when dealing with intricate motion patterns and occlusions. Current techniques struggle to handle occlusions and correctly capture complex motion patterns. Furthermore, these techniques could be computationally expensive, especially when dealing with big amounts of video data. An integrated strategy combining optical flow estimation with 3D convolutional neural network (3D-CNN) architectures is presented to address these limitations and boost motion detection systems' accuracy and efficiency. The suggested technique is unusual as it combines 3D CNNs with optical flow estimates. Motion vectors representing dynamic scene changes are produced by optical flow estimates, and spatiotemporal characteristics are extracted by 3D CNNs processing together with optical flow information. By using the complementing capabilities of both approaches, the method performs better in motion detection applications such as object tracking, action recognition, and anomaly detection in video streams. The efficiency of the strategy is assessed by experiments carried out on benchmark datasets. The results show that it is more accurate, resistant against occlusions, and computationally efficient than existing approaches. The suggested approach offers a viable way to improve motion detection capabilities for a range of practical uses. The results show a 98% improvement in processing efficiency and motion detection accuracy when compared to baseline methods. More research and advancement in this area are made possible by the attainable way that MATLAB's suggested approach offers to enhance motion detection skills in a variety of real-world applications.

A spatio-temporal model for violence detection based on spatial and temporal attention modules and 2D CNNs

A spatio-temporal model for violence detection based on spatial and temporal attention modules and 2D CNNs

Automated Liver Tumor Segmentation in DCE-MRI with 4D Deep Learning Integrating 3D CNNS and ConvLSTM

A state-of-the-art method for automatically segmenting liver tumours using Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) is shown in this study. This study is significant because it uses a 4D information deep learning model to tackle the hard problem of liver tumor segmentation. A combination of 3D CNNs and ConvLSTM networks, specifically built to capture spatial and temporal information inside the dynamic imaging sequence of DCE-MRI, is what the suggested model is all about. Utilizing diffusion-computed tomography (DCE-MRI) gives a lot of information vital for precise tumor segmentation by providing a complete picture of the vascular dynamics in the liver. The model makes use of spatial and temporal elements by combining 3D Convolutional Neural Networks (CNNs) with ConvLSTM networks; this allows for a more detailed comprehension of the changes that are happening over time. To overcome the difficulties caused by the constantly changing nature of DCE-MRI data, this integration of 4D information greatly improves the accuracy and consistency of liver tumor segmentation. Implementing and optimizing the suggested deep learning model are the main goals of this work. The training and calibration of the model to accurately capture liver tumor characteristics in dynamic imaging sequences is of utmost importance.

Improved Fault Diagnosis of Roller Bearings Using an Equal-Angle Integer-Period Array Convolutional Neural Network

This article presents a technique to carry out fault classification using an equal-angle integer-period array convolutional neural network (EAIP-CNN) to process the electrostatic signal of working roller bearings. Firstly, electrostatic signals were collected using uniform angle sampling to ensure the angle intervals between two adjacent data points stayed the same and the signal length was fixed to a pre-determined number of rotation cycles. Then, this one-dimensional signal was transformed into a two-dimensional matrix, where the component of each row was the signal in one period, and the ordinate value of each row represented the corresponding rotation period. Therefore, the row and column indexes of the matrix had a specific meaning instead of simply splitting and stacking the data. Finally, the matrixes were utilized to train the CNN network and test the classification performance. The results show that the classification rate using this technique reaches 95.6%, which is higher than that of 2D CNNs without equal-angle integer-period arrays.

CTM: Cross-time temporal module for fine-grained action recognition

Dynamic contextual attribute information in the time dimension is the key to fine-grained action recognition. Temporal contextual relationships cannot be captured by conventional 2D CNNs; good local time can be obtained by the 3D CNNs, but the 3D CNNs are computationally intensive and lack capability for global time. A parallel cross-time temporal module-CTM is proposed in this article, which aims to efficiently capture dynamic contextual information of both local and global temporal dimensions. With our study, we think that the 2D CNNs can better mine temporal features to enrich the contextual relationships of temporal dimensions. The CTM can be embedded into any existing 2D CNNs baseline in a plug-and-play manner, yielding a feature framework that can capture complex spatio-temporal modeling (CTNet) with a tiny additional computational cost. In the extensive validation experiments on three datasets(i.e., SomethingV1&V2, Jester, Diving48), both action recognition accuracy and runtime inference speed are obviously better than existing temporal contextual baseline optimization schemes with similar computational cost complexity, when the CTM embedded into any 2D CNNs framework to enhance the baseline.

3D Convolutional Neural Networks for Predicting Protein Structure for Improved Drug Recommendation

INTRODUCTION: Protein structure prediction is critical for recommendation personalized medicine and drug discovery. This paper introduces a robust approach using 3D Convolution Neural Networks (3D CNN’s) to improve the accuracy of the structure of protein structure thus contributing for the drug recommendation system.
 OBJECTIVES: In contrast to conventional techniques, 3D CNNs are able to identify complicated folding patterns and comprehend the subtle interactions between amino acids because they are able to capture spatial dependencies inside protein structures.
 METHODS: Data sets are collected from Protein Data Bank, including experimental protein structures and the drugs that interact with them, are used to train the model. With the efficient processing of three-dimensional data, the 3D CNNs exhibit enhanced capability in identifying minute structural details that are crucial for drug binding. This drug recommendation system novel method makes it easier to find potential drugs that interact well with particular protein structures.
 RESULTS: The performance of the proposed classifier is compared with the existing baseline methods with various parameters accuracy, precision, recall, F1 score, mean squared error (MSE) and area under the receiver operating characteristic curve (AUC-ROC).
 CONCLUSION: Deep learning and 3D structural insights work together to create a new generation of tailored and focused therapeutic interventions by speeding up the drug development process and improving the accuracy of pharmacological recommendations.

3D Convolutional Neural Networks based Movement Evaluation System for Gymnasts in Computer Vision Applications

This study introduces an innovative Movement Evaluation System, which utilizes state of art 3D CNNs in grassland of computer vision. This system presents a significant advancement in analyzing and assessing complex gymnastic movements, providing a comprehensive understanding of both spatial and temporal dynamics. By incorporating pose estimation algorithms, it accurately identifies body joints and positions to extract detailed spatial features. The 3D CNN further captures the temporal evolution of these features, enabling a precise analysis of the fluidity, rhythm, and synchronization of gymnastic movements over time. The effectiveness of this system is evaluated through performance metrics such as precision, recall, and accuracy. Aimed at enhancing sports science and athlete training, this research offers coaches, judges, and gymnasts a sophisticated tool for objective and standardized movement assessments. It has the potential to streamline the evaluation process in gymnastics and provide constructive feedback efficiently. The integration of 3D CNNs in this system marks a paradigm shift in utilizing advanced computer vision techniques to improve the understanding and refinement of gymnastic performances.

3D convolutional neural networks predict cellular metabolic pathway use from fluorescence lifetime decay data.

Fluorescence lifetime imaging of the co-enzyme reduced nicotinamide adenine dinucleotide (NADH) offers a label-free approach for detecting cellular metabolic perturbations. However, the relationships between variations in NADH lifetime and metabolic pathway changes are complex, preventing robust interpretation of NADH lifetime data relative to metabolic phenotypes. Here, a three-dimensional convolutional neural network (3D CNN) trained at the cell level with 3D NAD(P)H lifetime decay images (two spatial dimensions and one time dimension) was developed to identify metabolic pathway usage by cancer cells. NADH fluorescence lifetime images of MCF7 breast cancer cells with three isolated metabolic pathways, glycolysis, oxidative phosphorylation, and glutaminolysis were obtained by a multiphoton fluorescence lifetime microscope and then segmented into individual cells as the input data for the classification models. The 3D CNN models achieved over 90% accuracy in identifying cancer cells reliant on glycolysis, oxidative phosphorylation, or glutaminolysis. Furthermore, the model trained with human breast cancer cell data successfully predicted the differences in metabolic phenotypes of macrophages from control and POLG-mutated mice. These results suggest that the integration of autofluorescence lifetime imaging with 3D CNNs enables intracellular spatial patterns of NADH intensity and temporal dynamics of the lifetime decay to discriminate multiple metabolic phenotypes. Furthermore, the use of 3D CNNs to identify metabolic phenotypes from NADH fluorescence lifetime decay images eliminates the need for time- and expertise-demanding exponential decay fitting procedures. In summary, metabolic-prediction CNNs will enable live-cell and in vivo metabolic measurements with single-cell resolution, filling a current gap in metabolic measurement technologies.

Unified Spatio-Temporal Attention Models for Advanced Human Action Recognition & Detection

Abstract: This research paper introduces innovative approaches to enhance human action analytics in computer vision through the development of spatial and temporal attention models. The first model is based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units, aiming to extract discriminative spatial and temporal features from skeleton data for improved human action recognition and detection. The model selectively focuses on discriminative joints within each frame, utilizing a regularized cross-entropy loss for effective training. Additionally, a joint training strategy is proposed to optimize the learning process. The second model, a spatio-temporal attention (STA) network, is designed to address the limitations of existing 3D Convolutional Neural Networks (3D CNNs) in treating all video frames equally. The STA network characterizes beneficial information at both the frame and channel levels, leveraging differences in spatial and temporal dimensions to enhance the learning capability of 3D convolutions. The proposed STA method can be seamlessly integrated into state-of-the-art 3D CNN architectures for video action detection and recognition. Evaluation of diverse datasets, including SBU Kinect Interaction, NTU RGB+D, PKU-MMD, UCF-101, HMDB-51, and THUMOS 2014, demonstrates the effectiveness of both proposed models in achieving state-of-the-art performance for action recognition and detection tasks. The spatial and temporal attention models contribute significantly to capturing discriminative features, showcasing their potential in advancing the field of human action analytics in computer vision.

Deep Learning Approaches for Imaging-Based Automated Segmentation of Tuberous Sclerosis Complex

The present study presents a novel approach for identifying epileptogenic tubers in patients with tuberous sclerosis complex (TSC) and automating tuber segmentation using a three-dimensional convolutional neural network (3D CNN). The study retrospectively included 31 TSC patients whose lesions were manually annotated from multiparametric neuroimaging data. Epileptogenic tubers were determined via presurgical evaluation and stereoelectroencephalography recording. Neuroimaging metrics were extracted and compared between epileptogenic and non-epileptogenic tubers. Additionally, five datasets with different preprocessing strategies were used to construct and train 3D CNNs for automated tuber segmentation. The normalized positron emission tomography (PET) metabolic value was significantly lower in epileptogenic tubers defined via presurgical evaluation (p = 0.001). The CNNs showed high performance for localizing tubers, with an accuracy between 0.992 and 0.994 across the five datasets. The automated segmentations were highly correlated with clinician-based features. The neuroimaging characteristics for epileptogenic tubers were demonstrated, increasing surgical confidence in clinical practice. The validated deep learning detection algorithm yielded a high performance in determining tubers with an excellent agreement with reference clinician-based segmentation. Collectively, when coupled with our investigation of minimal input requirements, the approach outlined in this study represents a clinically invaluable tool for the management of TSC.

Gestures recognition based on multimodal fusion by using 3D CNNs

Gestures have long been recognized as an interaction technique that can provide a more natural, creative, and intuitive way to communicate with computers. However, some existing difficulties include the high probability that the same type of movement done at different speeds will be recognized as a different category of movement; cluttered, occluded, and low-resolution backgrounds; and the near-impossibility of fusing different types of features. To this end, we propose a novel framework for integrating different scales of RGB and motion skeletons to obtain higher recognition accuracy using multiple features. Specifically, we provide a network architecture that combines a three-dimensional convolutional neural network (3DCNN) and post-fusion to better embed different features. Also, we combine RGB and motion skeleton information at different scales to mitigate speed and background issues. Experiments on several gesture recognition public datasets show desirable results, validating the superiority of the proposed gesture recognition method. Finally, we do a human-computer interaction experiment to prove its practicality.