Deep 3D Research Articles

3D convolutional deep learning for nonlinear estimation of body composition from whole body morphology

Body composition prediction from 3D optical imagery has previously been studied with linear algorithms. In this study, we present a novel application of deep 3D convolutional graph networks and nonlinear Gaussian process regression for human body shape parameterization and body composition estimation. We trained and tested linear and nonlinear models with ablation studies on a novel ensemble body shape dataset containing 4286 scans. Nonlinear GPR produced up to a 20% reduction in prediction error and up to a 30% increase in precision over linear regression for both sexes in 10 tested body composition variables. Deep shape features produced 6–8% reduction in prediction error over linear PCA features for males only, and a 4–14% reduction in precision error for both sexes. All coefficients of determination (R2) for all predicted variables were above 0.86 and achieved lower estimation RMSEs than all previous work on 10 metrics of body composition.

Calorie detection in dishes based on deep learning and 3D reconstruction

Calorie detection in dishes based on deep learning and 3D reconstruction

Impact of Deep Learning 3D CT Super-Resolution on AI-Based Pulmonary Nodule Characterization

Background/Objectives: Correct pulmonary nodule volumetry and categorization is paramount for accurate diagnosis in lung cancer screening programs. CT scanners with slice thicknesses of multiple millimetres are still common worldwide, and slice thickness has an adverse effect on the accuracy of the pulmonary nodule volumetry. Methods: We propose a deep learning based super-resolution technique to generate thin-slice CT images from thick-slice CT images. Analysis of the lung nodule volumetry and categorization accuracy was performed using commercially available AI-based lung cancer screening software. Results: The accuracy of pulmonary nodule categorization increased from 72.7 percent to 94.5 percent when thick-slice CT images were converted to generated-thin-slice CT images. Conclusions: Applying the super-resolution-based slice generation on thick-slice CT images prior to automatic nodule evaluation significantly increases the accuracy of pulmonary nodule volumetry and corresponding pulmonary nodule category.

Fast 3D Transmission Tower Detection Based on Virtual Views

Advanced remote sensing technologies leverage extensive synthetic aperture radar (SAR) satellite data and high-resolution airborne light detection and ranging (LiDAR) data to swiftly capture comprehensive 3D information about electrical grid assets and their surrounding environments. This facilitates in-depth scene analysis for target detection and classification, allowing for the early recognition of potential hazards near transmission towers (TTs). These innovations present a groundbreaking strategy for the automated inspection of electrical grid assets. However, traditional 3D target detection techniques, which involve searching the entire 3D space, are marred by low accuracy and high computational demands. Although deep learning-based 3D target detection methods have significantly improved detection precision, they rely on a large volume of 3D target samples for training and are sensitive to point cloud data density. Moreover, these methods demonstrate low detection efficiency, constraining their application in the automated monitoring of electricity networks. This paper proposes a fast 3D target detection method using virtual views to overcome these challenges related to detection accuracy and efficiency. The method first utilizes cutting-edge 2D splatting technology to project 3D point clouds with diverse densities from a specific viewpoint, generating a 2D virtual image. Then, a novel local–global dual-path feature fusion network based on YOLO is applied to detect TTs on the virtual image, ensuring efficient and accurate identification of their positions and types. Finally, by leveraging the projection transformation between the virtual image and the 3D point cloud, combined with a 3D region growing algorithm, the 3D points belonging to the TTs are extracted from the whole 3D point cloud. The effectiveness of the proposed method in terms of target detection rate and efficiency is validated through experiments on synthetic datasets and outdoor LiDAR point clouds.

Deep learning-based polarization 3D imaging method for underwater targets.

The significant absorption and scattering of light during its propagation in water severely degrade the quality of underwater imaging, presenting challenges for developing high-precision 3D imaging techniques based on optical methods. Polarization imaging has demonstrated effectiveness in mitigating the effects of scattering, making it a valuable approach for underwater imaging. Additionally, the polarization state of reflected light can be utilized for surface normal estimation and 3D shape reconstruction. This paper presents a learning-based method for 3D shape reconstruction of underwater targets using shape from polarization techniques. To address the lack of publicly available datasets for underwater polarization 3D imaging, we have developed a data acquisition system that simulates Jerlov Type I water conditions, creating a dataset of underwater polarized images along with corresponding ground truth surface normal images. Furthermore, we propose a network framework based on Attention U2Net for the 3D reconstruction of underwater polarized images. This framework is designed to capture detailed texture information of underwater targets and incorporates an effective polarization representation to resolve azimuthal ambiguity, thus enhancing the accuracy of underwater 3D imaging. Experimental results demonstrate that our method effectively addresses azimuthal ambiguity, reduces texture loss during reconstruction, and improves the accuracy of surface normal estimation, achieving superior performance compared to existing methods.

Unsupervised Anomaly Detection for Improving Adversarial Robustness of 3D Object Detection Models

Three-dimensional object detection based on deep neural networks (DNNs) is widely used in safety-related applications, such as autonomous driving. However, existing research has shown that 3D object detection models are vulnerable to adversarial attacks. Hence, the improvement on the robustness of deep 3D detection models under adversarial attacks is investigated in this work. A deep autoencoder-based anomaly detection method is proposed, which has a strong ability to detect elaborate adversarial samples in an unsupervised way. The proposed anomaly detection method operates on a given Light Detection and Ranging (LiDAR) scene in its Bird’s Eye View (BEV) image and reconstructs the scene through an autoencoder. To improve the performance of the autoencoder, an augmented memory module with typical normal patterns recorded is introduced. It is designed to help the model to amplify the reconstruction errors of malicious samples with normal samples negligibly affected. Experiments on several public datasets show that the proposed anomaly detection method achieves an AUC of 0.8 under adversarial attacks and improves the robustness of 3D object detection.

Improved Consistency of Lung Nodule Categorization in CT Scans with Heterogeneous Slice Thickness by Deep Learning-Based 3D Super-Resolution

Background/Objectives: Accurate volumetric assessment of lung nodules is an essential element of low-dose lung cancer screening programs. Current guidance recommends applying specific thresholds to measured nodule volume to make the following clinical decisions. In reality, however, CT scans often have heterogeneous slice thickness which is known to adversely impact the accuracy of nodule volume assessment. Methods: In this study, a deep learning (DL)-based 3D super-resolution method is proposed for generating thin-slice CT images from heterogeneous thick-slice CT images in lung cancer screening. We evaluated the performance in a qualitative way by radiologist’s perceptual assessment as well as in a quantitative way by accuracy of nodule volume measurements and agreement of volume-based Lung-RADS nodule category. Results: A 5-point Likert scale tabulated by two radiologists showed that the quality of DL-generated thin-slice images from thick-slice CT images were on a par with the image quality of ground truth thin-slice CT images. Furthermore, thick- and thin-slice CT images had a nodule volume difference of 52.2 percent on average which was reduced to a 15.7 percent difference with DL-generated thin-slice CT. In addition, the proposed method increased the agreement of lung nodule categorization using Lung-RADS by 74 percent. Conclusions: The proposed DL approach for slice thickness normalization has a potential for improving the accuracy of lung nodule volumetry and facilitating more reliable early lung nodule detection.

Three-Dimensional Shape Reconstruction from Digital Freehand Design Sketching Based on Deep Learning Techniques

This paper proposes a method for 3D reconstruction from Freehand Design Sketching (FDS) in architecture and industrial design. The implementation begins by extracting features from the FDS using the self-supervised learning model DINO, followed by the continuous Signed Distance Function (SDF) regression as an implicit representation through a Multi-Layer Perceptron network. Taking eyeglass frames as an example, the 2D contour and freehand sketch optimize the alignment by their geometrical similarity while exploiting symmetry to improve reconstruction accuracy. Experiments demonstrate that this method can effectively reconstruct high-quality 3D models of eyeglass frames from 2D freehand sketches, outperforming existing deep learning-based 3D reconstruction methods. This research offers practical information for understanding 3D modeling methodology for FDS, triggering multiple modes of design creativity and efficient scheme adjustments in industrial or architectural conceptual design. In conclusion, this novel approach integrates self-supervised learning and geometric optimization to achieve unprecedented fidelity in 3D reconstruction from FDS, setting a new benchmark for AI-driven design processes in industrial and architectural applications.

Mixed reality infrastructure based on deep learning medical image segmentation and 3D visualization for bone tumors using DCU-Net.

Segmenting and reconstructing 3D models of bone tumors from 2D image data is of great significance for assisting disease diagnosis and treatment. However, due to the low distinguishability of tumors and surrounding tissues in images, existing methods lack accuracy and stability. This study proposes a U-Net model based on double dimensionality reduction and channel attention gating mechanism, namely the DCU-Net model for oncological image segmentation. After realizing automatic segmentation and 3D reconstruction of osteosarcoma by optimizing feature extraction and improving target space clustering capabilities, we built a mixed reality (MR) infrastructure and explored the application prospects of the infrastructure combining deep learning-based medical image segmentation and mixed reality in the diagnosis and treatment of bone tumors. We conducted experiments using a hospital dataset for bone tumor segmentation, used the optimized DCU-Net and 3D reconstruction technology to generate bone tumor models, and used set similarity (DSC), recall (R), precision (P), and 3D vertex distance error (VDE) to evaluate segmentation performance and 3D reconstruction effects. Then, two surgeons conducted clinical examination experiments on patients using two different methods, viewing 2D images and virtual reality infrastructure, and used the Likert scale (LS) to compare the effectiveness of surgical plans of the two methods. The DSC, R and P values of the model introduced in this paper all exceed 90%, which has significant advantages compared with methods such as U-Net and Attention-Uet. Furthermore, LS showed that clinicians in the DCU-Net-based MR group had better spatial awareness of tumor preoperative planning. The deep learning DCU-Net algorithm model can improve the performance of tumor CT image segmentation, and the reconstructed fine model can better reflect the actual situation of individual tumors; the MR system constructed based on this model enhances clinicians' understanding of tumor morphology and spatial relationships. The MR system based on deep learning and three-dimensional visualization technology has great potential in the diagnosis and treatment of bone tumors, and is expected to promote clinical practice and improve efficacy.

A Review of 3D Human Pose Estimation

Abstract. Visual perception and human body recognition are fundamental capabilities required for effective and safe interactions between artificial intelligence (AI), computer vision, and humans in real-world scenarios. Recent groundbreaking developments in AI and computer vision have resulted in major advancements in human body recognition technology. However, research in human body recognition is still in the early stages of the product lifecycle. Identifying the three-dimensional locations of the joints in the human body from pictures or videos is known as 3D posture estimation. Although it is widely used in areas like human motion analysis and robotics, it continues to be a difficult task due to challenges such as depth ambiguity and the scarcity of robust datasets. Over the past decade, numerous methods have been developed, many of which are based on deep learning, significantly improving the performance of existing benchmarks. A comprehensive literature review of this field is crucial for future development. However, in nowadays,more and more such research has mainly concentrated on traditional techniques, requirement for a comprehensive examination of tools based on deep learning. This paper delivers a thorough overview of current deep learning-based 3D pose estimation algorithms, outlining their advantages and limitations while providing a detailed understanding of the field. It also explores commonly used benchmark datasets and methods for analyzing human poses in unlabeled field images, providing a thorough comparative analysis. Finally, insights are provided to aid in the design of future models and algorithms.

Contextual Attribution Maps-Guided Transferable Adversarial Attack for 3D Object Detection

The study of LiDAR-based 3D object detection and its robustness under adversarial attacks has achieved great progress. However, existing adversarial attack methods mainly focus on the targeted object, which destroys the integrity of the object and makes the attack easy to perceive. In this work, we propose a novel adversarial attack against deep 3D object detection models named the contextual attribution maps-guided attack (CAMGA). Based on the combinations of subregions in the context area and their impact on the prediction results, contextual attribution maps can be generated. An attribution map exposes the influence of individual subregions in the context area on the detection results and narrows down the scope of the adversarial attack. Subsequently, perturbations are generated under the guidance of a dual loss, which is proposed to suppress the detection results and maintain visual imperception simultaneously. The experimental results proved that the CAMGA method achieved an attack success rate of over 68% on three large-scale datasets and 83% on the KITTI dataset. Meanwhile, the CAMGA has a transfer attack success rate of at least 50% against all four victim detectors, as they all overly rely on contextual information.

Multivariate Brain Tumor Detection in 3D-MRI Images Using Optimised Segmentation and Unified Classification Model.

3D Magnetic Resonance Imaging (3D-MRI) analysis of brain tumours is an important tool for gathering information needed for diagnosis and disease therapy planning. However, during the brain tumor segmentation process existing techniques have segmentation error while identifying tumor location and extended tumor regions due to improper extraction of initial contour points and overlapping tissue intensity distributions. Hence a novel Duo-step optimised Pyramidal SegNet has been proposed in which multiscale contrast convolutional attention module improve contrast and the tumor edge has been extracted based on location and tumor extension using Duo-step darning needle optimisation that set initial contour points and pyramidal level set segmentation with ancillary Sobel edge operator extract the tumour region from all 2D MRI image slices without having overlapped tissue intensity distributions thereby effectively minimises segmentation error. Furthermore, during the classification of segmented tumor region based on its type, irregular planimetric volume and low interrater concordance of multivariate brain tumors reduce the detection rate due to neglecting the extraction of contextual and symmetric features. Hence 3D brain Unified NN has been proposed in which adaptive multi-layer deep unified encoder module extract 3D contextual and symmetric features by measuring the difference from the observed region and contralateral region and the multivariate brain tumors are classified with boosted Sparse Categorical Cross entropy loss calculation to demonstrate high detection rate. The results obtained for the BraTS2020 and Brain Tumor Detection 2020 data sets showed that the proposed model outperforms existing techniques with excellent precision of 97%, 97.5%, recall of 99%, 97.8%, and accuracy of 95.7%, 98.4%, respectively.

DEGAN: Detail-Enhanced Generative Adversarial Network for Monocular Depth-Based 3D Reconstruction

Although deep networks-based 3D reconstruction methods can recover the 3D geometry given few inputs, they may produce unfaithful reconstruction when predicting occluded parts of 3D objects. To address the issue, we propose Detail-Enhanced Generative Adversarial Network (DEGAN) which consists of Encoder–Decoder-Based Generator (EDGen) and Voxel-Point Embedding Network-Based Discriminator (VPDis) for 3D reconstruction from a monocular depth image of an object. Firstly, EDGen decodes the features from the 2.5D voxel grid representation of an input depth image and generates the 3D occupancy grid under GAN losses and a sampling point loss. The sampling loss can improve the accuracy of predicted points with high uncertainty. VPDis helps reconstruct the details under voxel and point adversarial losses, respectively. Experimental results show that DEGAN not only outperforms several state-of-the-art methods on both public ModelNet and ShapeNet datasets but also predicts more reliable occluded/missing parts of 3D objects.

Performance of vibration and current signals in the fault diagnosis of induction motors using deep learning and machine learning techniques

Induction motors (IMs) play a pivotal role in various industrial applications, powering critical systems such as pumps, compressors, fans, blowers, and refrigeration and air conditioning systems. Monitoring the health of these IMs is essential for ensuring reliable operation. Numerous sensors, including vibration, current, temperature, acoustic, and power sensors, can be employed for their health monitoring. This article conducts a comprehensive comparative analysis of two widely used sensors—vibration and current, for classifying different health states of IMs, such as a healthy condition, bearing fault, and misalignment. The study employed deep learning techniques, specifically 1D and 2D convolutional neural networks, trained on raw data. Additionally, machine learning techniques, including random forest and XGBoost, were utilized and trained on features derived from preprocessed signals using fast Fourier transform and discrete wavelet decomposition. Comparative results indicated that vibration signals achieved remarkably high accuracy, nearly 100%, in detecting the investigated mechanical faults, while current signals, after signal processing and manual feature extraction, achieved an accuracy of 87.41%. These results demonstrate that, though current sensors are a viable alternative to vibration sensors, their performance can be affected by the type and degree of the considered faults. This study also highlights the attributes of vibration and current signals in the health monitoring of rotating machinery such as IMs.

From Methods to Applications: A Review of Deep 3D Human Motion Capture

From Methods to Applications: A Review of Deep 3D Human Motion Capture

Personalized Fitting of Respiratory Mask Using Deep Learning and 3D Facial Modeling

Personalized Fitting of Respiratory Mask Using Deep Learning and 3D Facial Modeling

Deep models for multi-view 3D object recognition: a review

This review paper focuses on the progress of deep learning-based methods for multi-view 3D object recognition. It covers the state-of-the-art techniques in this field, specifically those that utilize 3D multi-view data as input representation. The paper provides a comprehensive analysis of the pipeline for deep learning-based multi-view 3D object recognition, including the various techniques employed at each stage. It also presents the latest developments in CNN-based and transformer-based models for multi-view 3D object recognition. The review discusses existing models in detail, including the datasets, camera configurations, view selection strategies, pre-trained CNN architectures, fusion strategies, and recognition performance. Additionally, it examines various computer vision applications that use multi-view classification. Finally, it highlights future directions, factors impacting recognition performance, and trends for the development of multi-view 3D object recognition method.

CQformer: Learning Dynamics Across Slices in Medical Image Segmentation.

Prevalent studies on deep learning-based 3D medical image segmentation capture the continuous variation across 2D slices mainly via convolution, Transformer, inter-slice interaction, and time series models. In this work, via modeling this variation by an ordinary differential equation (ODE), we propose a cross instance query-guided Transformer architecture (CQformer) that leverages features from preceding slices to improve the segmentation performance of subsequent slices. Its key components include a cross-attention mechanism in an ODE formulation, which bridges the features of contiguous 2D slices of the 3D volumetric data. In addition, a regression head is employed to shorten the gap between the bottleneck and the prediction layer. Extensive experiments on 7 datasets with various modalities (CT, MRI) and tasks (organ, tissue, and lesion) demonstrate that CQformer outperforms previous state-of-the-art segmentation algorithms on 6 datasets by 0.44%-2.45%, and achieves the second highest performance of 88.30% on the BTCV dataset. The code will be publicly available after acceptance.

Bio-inspired feature selection for early diagnosis of Parkinson's disease through optimization of deep 3D nested learning.

Parkinson's disease (PD) is one of the most common neurodegenerative disorders that affect the quality of human life of millions of people throughout the world. The probability of getting affected by this disease increases with age, and it is common among the elderly population. Early detection can help in initiating medications at an earlier stage. It can significantly slow down the progression of this disease, assisting the patient to maintain a good quality of life for a more extended period. Magnetic resonance imaging (MRI)-based brain imaging is an area of active research that is used to diagnose PD disease early and to understand the key biomarkers. The prior research investigations using MRI data mainly focus on volume, structural, and morphological changes in the basal ganglia (BG) region for diagnosing PD. Recently, researchers have emphasized the significance of studying other areas of the human brain for a more comprehensive understanding of PD and also to analyze changes happening in brain tissue. Thus, to perform accurate diagnosis and treatment planning for early identification of PD, this work focuses on learning the onset of PD from images taken from whole-brain MRI using a novel 3D-convolutional neural network (3D-CNN) deep learning architecture. The conventional 3D-Resent deep learning model, after various hyper-parameter tuning and architectural changes, has achieved an accuracy of 90%. In this work, a novel 3D-CNN architecture was developed, and after several ablation studies, the model yielded results with an improved accuracy of 93.4%. Combining features from the 3D-CNN and 3D ResNet models using Canonical Correlation Analysis (CCA) resulted in 95% accuracy. For further enhancements of the model performance, feature fusion with optimization was employed, utilizing various optimization techniques. Whale optimization based on a biologically inspired approach was selected on the basis of a convergence diagram. The performance of this approach is compared to other methods and has given an accuracy of 97%. This work represents a critical advancement in improving PD diagnosis techniques and emphasizing the importance of deep nested 3D learning and bio-inspired feature selection.

Anomaly Detection in Video Surveillance: A Comparative Analysis of Deep Learning Models

Anomaly detection in video surveillance is critical for enhancing security and public safety across various applications, including traffic monitoring, public spaces, and industrial settings. Traditional methods often struggle with the complexity and variability of real-world data, prompting a shift towards advanced machine learning models. This paper presents a comprehensive analysis of deep learning algorithms, including YOLOv5, 3D CNNs, LSTM, Deep SVDD, Vision Transformers, Temporal Transformers, and Autoencoders, applied to three benchmark datasets: CIFAR-10, MVTec AD, and UCSD Anomaly Detection. We compare these algorithms based on accuracy, precision, recall, and F1-score, providing insight into their strengths and weaknesses. The results suggest that Vision Transformers and CNN-LSTM hybrids offer superior performance across spatial and temporal anomaly detection tasks.