Volumetric Convolutional Neural Network Research Articles

LVNet: A lightweight volumetric convolutional neural network for real-time and high-performance recognition of 3D objects

LVNet: A lightweight volumetric convolutional neural network for real-time and high-performance recognition of 3D objects

A Novel Approach to Skin Lesion Segmentation: Multipath Fusion Model with Fusion Loss.

Segmentation of skin lesions plays a very important role in the early detection of skin cancer. However, indistinguishability due to various artifacts such as hair and contrast between normal skin and lesioned skin is an important challenge for specialist dermatologists. Computer-aided diagnostic systems using deep convolutional neural networks are gaining importance in order to cope with difficulties. This study focuses on deep learning-based fusion networks and fusion loss functions. For the automatic segmentation of skin lesions, U-Net (U-Net + ResNet 2D) with 2D residual blocks and 2D volumetric convolutional neural networks were fused for the first time in this study. Also, a new fusion loss function is proposed by combining Dice Loss (DL) and Focal Tversky Loss (FTL) to make the proposed fused model more robust. Of the 2594 image dataset, 20% is reserved for test data and 80% for training data. In test data training, a Jaccard score of 0.837 and a dice score of 0.918 were obtained. The proposed model was also scored on the ISIC 2018 Task 1 test images, whose ground truths were not shared. The proposed model performed well and achieved a Jaccard index of 0.800 and a dice score of 0.880 in the ISIC 2018 Task 1 test set. In addition, it has been observed that the new fused loss function obtained by fusing Focal Tversky Loss and Dice Loss functions in the proposed model increases the robustness of the model in the tests. The proposed new loss function fusion model has outstripped the cutting-edge approaches in the literature.

End-to-End Deep Learning Architectures Using 3D Neuroimaging Biomarkers for Early Alzheimer’s Diagnosis

This study uses magnetic resonance imaging (MRI) data to propose end-to-end learning implementing volumetric convolutional neural network (CNN) models for two binary classification tasks: Alzheimer’s disease (AD) vs. cognitively normal (CN) and stable mild cognitive impairment (sMCI) vs. AD. The baseline MP-RAGE T1 MR images of 245 AD patients and 229 with sMCI were obtained from the ADNI dataset, whereas 245 T1 MR images of CN people were obtained from the IXI dataset. All of the images were preprocessed in four steps: N4 bias field correction, denoising, brain extraction, and registration. End-to-end-learning-based deep CNNs were used to discern between different phases of AD. Eight CNN-based architectures were implemented and assessed. The DenseNet264 excelled in both types of classification, with 82.5% accuracy and 87.63% AUC for training and 81.03% accuracy for testing relating to the sMCI vs. AD and 100% accuracy and 100% AUC for training and 99.56% accuracy for testing relating to the AD vs. CN. Deep learning approaches based on CNN and end-to-end learning offer a strong tool for examining minute but complex properties in MR images which could aid in the early detection and prediction of Alzheimer’s disease in clinical settings.

Skin Lesion Segmentation Based on Edge Attention Vnet with Balanced Focal Tversky Loss

Segmentation of skin lesions from dermoscopic images plays an essential role in the early detection of skin cancer. However, skin lesion segmentation is still challenging due to artifacts such as indistinguishability between skin lesion and normal skin, hair on the skin, and reflections in the obtained dermoscopy images. In this study, an edge attention network (ET-Net) combining edge guidance module (EGM) and weighted aggregation module is added to the 2D volumetric convolutional neural network (Vnet 2D) to maximize the performance of skin lesion segmentation. In addition, the proposed fusion model presents a new fusion loss function by combining balanced binary cross-entropy (BBCE) and focal Tversky loss (FTL). The proposed model has been tested on the ISIC 2018 Task 1 Lesion Boundary Segmentation Challenge dataset. The proposed model outperformed the state-of-the-art studies as a result of the tests.

AlzVNet: A volumetric convolutional neural network for multiclass classification of Alzheimer’s disease through multiple neuroimaging computational approaches

Alzheimer's disease is a degenerative neurological disease that causes a loss of cognitive skills and has no known treatment. Alzheimer's disease (AD) must be detected early, before symptoms appear, in order to be treated effectively. In this study, we used a deep learning approach called a convolutional neural network to classify Alzheimer's disease into three categories using a neuroimaging biomarker called T1w-MRI. Our research is the first to look at the results of three neuroimaging computational approaches in a systematic way (3D subject-level, 3D patch-based and slice-based). To show Alzheimer detection using deep convolutional neural networks, three distinct Slice Based methods are used (Subset selection method, uniform selection method, Interpolation zoom method). For 3D patch-based approaches, we investigated the classification accuracy of distinct non-overlapping patches ranging in size from small to medium to large (from 32, 40, 48, 56, 64, 72, 80, till 88). Our findings revealed that 1) our 3-class classification model performed best, with 98.3 percent accuracy percent (highest accuracy obtained until now as per our best knowledge); 2) The 3D Subject-level approach was the most efficient, followed by 3D-patch-based and then Slice-based approaches, with classification accuracy of 98.26 percent, 97.48 percent, and 95.40 percent, respectively; and 3) The same network had the most accuracy for bigger patches (size 72, 80, 88), followed by medium-sized (size 56, 64) to tiny patches (size 32, 40, 48). Large patches had a classification accuracy of 97.48 percent, while medium patches had a classification accuracy of 96.62 percent, and small patches had an accuracy of 86.49 percent. 4)) Even slice selection and interpolation selection exceeded subset slice selection with three-class classification accuracy of 95.37 percent and 94.57 percent, respectively, compared to 92.57 percent for subset slice selection.

Classification and Visualization of Chemotherapy-Induced Cognitive Impairment in Volumetric Convolutional Neural Networks.

Breast cancer is the most common female cancer worldwide, and breast cancer accounts for 30% of female cancers. Of all the treatment modalities, breast cancer survivors who have undergone chemotherapy might complain about cognitive impairment during and after cancer treatment. This phenomenon, chemo-brain, is used to describe the alterations in cognitive functions after receiving systemic chemotherapy. Few reports detect the chemotherapy-induced cognitive impairment (CICI) by performing functional MRI (fMRI) and a deep learning analysis. In this study, we recruited 55 postchemotherapy breast cancer survivors (C+ group) and 65 healthy controls (HC group) and extracted mean fractional amplitudes of low-frequency fluctuations (mfALFF) from resting-state fMRI as our input feature. Two state-of-the-art deep learning architectures, ResNet-50 and DenseNet-121, were transformed to 3D, embedded with squeeze and excitation (SE) blocks and then trained to differentiate cerebral alterations based on the effect of chemotherapy. An integrated gradient was applied to visualize the pattern that was recognized by our model. The average performance of SE-ResNet-50 models was an accuracy of 80%, precision of 78% and recall of 70%; on the other hand, the SE-DenseNet-121 model reached identical results with an average of 80% accuracy, 86% precision and 80% recall. The regions with the greatest contributions highlighted by the integrated gradients algorithm for differentiating chemo-brain were the frontal, temporal, parietal and occipital lobe. These regions were consistent with other studies and strongly associated with the default mode and dorsal attention networks. We constructed two volumetric state-of-the-art models and visualized the patterns that are critical for identifying chemo-brains from normal brains. We hope that these results will be helpful in clinically tracking chemo-brain in the future.

A Deep Learning Approach for Segmentation, Classification, and Visualization of 3-D High-Frequency Ultrasound Images of Mouse Embryos.

Segmentation and mutant classification of high-frequency ultrasound (HFU) mouse embryo brain ventricle (BV) and body images can provide valuable information for developmental biologists. However, manual segmentation and identification of BV and body requires substantial time and expertise. This article proposes an accurate, efficient and explainable deep learning pipeline for automatic segmentation and classification of the BV and body. For segmentation, a two-stage framework is implemented. The first stage produces a low-resolution segmentation map, which is then used to crop a region of interest (ROI) around the target object and serve as the probability map of the autocontext input for the second-stage fine-resolution refinement network. The segmentation then becomes tractable on high-resolution 3-D images without time-consuming sliding windows. The proposed segmentation method significantly reduces inference time (102.36-0.09 s/volume ≈ 1000× faster) while maintaining high accuracy comparable to previous sliding-window approaches. Based on the BV and body segmentation map, a volumetric convolutional neural network (CNN) is trained to perform a mutant classification task. Through backpropagating the gradients of the predictions to the input BV and body segmentation map, the trained classifier is found to largely focus on the region where the Engrailed-1 (En1) mutation phenotype is known to manifest itself. This suggests that gradient backpropagation of deep learning classifiers may provide a powerful tool for automatically detecting unknown phenotypes associated with a known genetic mutation.

A 3-D Lightweight Convolutional Neural Network for Detecting Docking Structures in Cluttered Environments

Abstract The maritime industry has been following the paradigm shift toward the automation of typically intelligent procedures, with research regarding autonomous surface vehicles (ASVs) having seen an upward trend in recent years. However, this type of vehicle cannot be employed on a full scale until a few challenges are solved. For example, the docking process of an ASV is still a demanding task that currently requires human intervention. This research work proposes a volumetric convolutional neural network (vCNN) for the detection of docking structures from 3-D data, developed according to a balance between precision and speed. Another contribution of this article is a set of synthetically generated data regarding the context of docking structures. The dataset is composed of LiDAR point clouds, stereo images, GPS, and Inertial Measurement Unit (IMU) information. Several robustness tests carried out with different levels of Gaussian noise demonstrated an average accuracy of 93.34% and a deviation of 5.46% for the worst case. Furthermore, the system was fine-tuned and evaluated in a real commercial harbor, achieving an accuracy of over 96%. The developed classifier is able to detect different types of structures and works faster than other state-of-the-art methods that establish their performance in real environments.

OctSurf: Efficient hierarchical voxel-based molecular surface representation for protein-ligand affinity prediction

Voxel-based 3D convolutional neural networks (CNNs) have been applied to predict protein-ligand binding affinity. However, the memory usage and computation cost of these voxel-based approaches increase cubically with respect to spatial resolution and sometimes make volumetric CNNs intractable at higher resolutions. Therefore, it is necessary to develop memory-efficient alternatives that can accelerate the convolutional operation on 3D volumetric representations of the protein-ligand interaction. In this study, we implement a novel volumetric representation, OctSurf, to characterize the 3D molecular surface of protein binding pockets and bound ligands. The OctSurf surface representation is built based on the octree data structure, which has been widely used in computer graphics to efficiently represent and store 3D object data. Vanilla 3D-CNN approaches often divide the 3D space of objects into equal-sized voxels. In contrast, OctSurf recursively partitions the 3D space containing the protein-ligand pocket into eight subspaces called octants. Only those octants containing van der Waals surface points of protein or ligand atoms undergo the recursive subdivision process until they reach the predefined octree depth, whereas unoccupied octants are kept intact to reduce the memory cost. Resulting non-empty leaf octants approximate molecular surfaces of the protein pocket and bound ligands. These surface octants, along with their chemical and geometric features, are used as the input to 3D-CNNs. Two kinds of CNN architectures, VGG and ResNet, are applied to the OctSurf representation to predict binding affinity. The OctSurf representation consumes much less memory than the conventional voxel representation at the same resolution. By restricting the convolution operation to only octants of the smallest size, our method also alleviates the overall computational overhead of CNN. A series of experiments are performed to demonstrate the disk storage and computational efficiency of the proposed learning method. Our code is available at the following GitHub repository: https://github.uconn.edu/mldrugdiscovery/OctSurf.

Advancing Autonomous Surface Vehicles: A 3D Perception System for the Recognition and Assessment of Docking-Based Structures

The automation of typically intelligent and decision-making processes in the maritime industry leads to fewer accidents and more cost-effective operations. However, there are still lots of challenges to solve until fully autonomous systems can be employed. Artificial Intelligence (AI) has played a major role in this paradigm shift and shows great potential for solving some of these challenges, such as the docking process of an autonomous vessel. This work proposes a lightweight volumetric Convolutional Neural Network (vCNN) capable of recognizing different docking-based structures using 3D data in real-time. A synthetic-to-real domain adaptation approach is also proposed to accelerate the training process of the vCNN. This approach makes it possible to greatly decrease the cost of data acquisition and the need for advanced computational resources. Extensive experiments demonstrate an accuracy of over 90% in the recognition of different docking structures, using low resolution sensors. The inference time of the system was about 120ms on average. Results obtained using a real Autonomous Surface Vehicle (ASV) demonstrated that the vCNN trained with the synthetic-to-real domain adaptation approach is suitable for maritime mobile robots. This novel AI recognition method, combined with the utilization of 3D data, contributes to an increased robustness of the docking process regarding environmental constraints, such as rain and fog, as well as insufficient lighting in nighttime operations.

Automated glioma grading on conventional MRI images using deep convolutional neural networks.

Gliomas are the most common primary tumor of the brain and are classified into grades I-IV of the World Health Organization (WHO), based on their invasively histological appearance. Gliomas grading plays an important role to determine the treatment plan and prognosis prediction. In this study we propose two novel methods for automatic, non-invasively distinguishing low-grade (Grades II and III) glioma (LGG) and high-grade (grade IV) glioma (HGG) on conventional MRI images by using deep convolutional neural networks (CNNs). All MRI images have been preprocessed first by rigid image registration and intensity inhomogeneity correction. Both proposed methods consist of two steps: (a) three-dimensional (3D) brain tumor segmentation based on a modification of the popular U-Net model; (b) tumor classification on segmented brain tumor. In the first method, the slice with largest area of tumor is determined and the state-of-the-art mask R-CNN model is employed for tumor grading. To improve the performance of the grading model, a two-dimensional (2D) data augmentation has been implemented to increase both the amount and the diversity of the training images. In the second method, denoted as 3DConvNet, a 3D volumetric CNNs is applied directly on bounding image regions of segmented tumor for classification, which can fully leverage the 3D spatial contextual information of volumetric image data. The proposed schemes were evaluated on The Cancer Imaging Archive (TCIA) low grade glioma (LGG) data, and the Multimodal Brain Tumor Image Segmentation (BraTS) Benchmark 2018 training datasets with fivefold cross validation. All data are divided into training, validation, and test sets. Based on biopsy-proven ground truth, the performance metrics of sensitivity, specificity, and accuracy are measured on the test sets. The results are 0.935 (sensitivity), 0.972 (specificity), and 0.963 (accuracy) for the 2D Mask R-CNN based method, and 0.947 (sensitivity), 0.968 (specificity), and 0.971 (accuracy) for the 3DConvNet method, respectively. In regard to efficiency, for 3D brain tumor segmentation, the program takes around ten and a half hours for training with 300 epochs on BraTS 2018 dataset and takes only around 50s for testing of a typical image with a size of 160×216×176. For 2D Mask R-CNN based tumor grading, the program takes around 4h for training with around 60000 iterations, and around 1s for testing of a 2D slice image with size of 128×128. For 3DConvNet based tumor grading, the program takes around 2h for training with 10000 iterations, and 0.25s for testing of a 3D cropped image with size of 64×64×64, using a DELL PRECISION Tower T7910, with two NVIDIA Titan Xp GPUs. Two effective glioma grading methods on conventional MRI images using deep convolutional neural networks have been developed. Our methods are fully automated without manual specification of region-of-interests and selection of slices for model training, which are common in traditional machine learning based brain tumor grading methods. This methodology may play a crucial role in selecting effective treatment options and survival predictions without the need for surgical biopsy.

Auto-ORVNet: Orientation-Boosted Volumetric Neural Architecture Search for 3D Shape Classification

Recently, more and more 3D shape datasets have become publicly available and significant results have been attained in 3D shape classification with 3D volumetric convolutional neural networks. However, the existing 3D volumetric networks have a problem with balancing model scale and classification accuracy. To address this problem, neural architecture search (NAS) was introduced into 3D shape classification tasks to search for a model satisfying both requirements. Automatically generating neural networks under NAS has attracted increasing research interest in recent years. The models learned by NAS outperform many manually designed networks in several 2D tasks like image classification, detection and semantic segmentation. In this paper, the differentiable formulation of NAS is exploited to search for several repeatable computation cells. The introduction of many light-weight designs for 3D CNNs assists in the construction of deep models with fewer parameters. The loss for the classification task along with the loss for orientation prediction are combined to guide such search. Extensive experiments are designed to evaluate candidate models on three datasets. The results demonstrate that without any pretraining, our discovered model for 3D shape classification outperforms most manually designed networks with small parameter sizes, whilst also showing that our model achieves a balance between model scale and classification accuracy.

Classification and Visualization of Alzheimer\u2019s Disease using Volumetric Convolutional Neural Network and Transfer Learning

Recently, deep-learning-based approaches have been proposed for the classification of neuroimaging data related to Alzheimer’s disease (AD), and significant progress has been made. However, end-to-end learning that is capable of maximizing the impact of deep learning has yet to receive much attention due to the endemic challenge of neuroimaging caused by the scarcity of data. Thus, this study presents an approach meant to encourage the end-to-end learning of a volumetric convolutional neural network (CNN) model for four binary classification tasks (AD vs. normal control (NC), progressive mild cognitive impairment (pMCI) vs. NC, stable mild cognitive impairment (sMCI) vs. NC and pMCI vs. sMCI) based on magnetic resonance imaging (MRI) and visualizes its outcomes in terms of the decision of the CNNs without any human intervention. In the proposed approach, we use convolutional autoencoder (CAE)-based unsupervised learning for the AD vs. NC classification task, and supervised transfer learning is applied to solve the pMCI vs. sMCI classification task. To detect the most important biomarkers related to AD and pMCI, a gradient-based visualization method that approximates the spatial influence of the CNN model’s decision was applied. To validate the contributions of this study, we conducted experiments on the ADNI database, and the results demonstrated that the proposed approach achieved the accuracies of 86.60% and 73.95% for the AD and pMCI classification tasks respectively, outperforming other network models. In the visualization results, the temporal and parietal lobes were identified as key regions for classification.

1.2 Watt Classification of 3D Voxel Based Point-clouds using a CNN on a Neural Compute Stick

With the recent surge in popularity of Convolutional Neural Networks (CNNs), motivated by their significant performance in many classification and related tasks, a new challenge now needs to be addressed: how to accommodate CNNs in mobile devices, such as drones, smartphones, and similar low-power devices? In order to tackle this challenge we exploit the Vision Processing Unit (VPU) that combines dedicated CNN hardware blocks and very low power requirement. The lack of readily available training data and memory requirements are two of the factors hindering the training and accuracy performance of 3D CNNs. In this paper, we propose a method for generating synthetic 3D point-clouds from realistic CAD scene models to enrich the training process for volumetric CNNs. Furthermore, an efficient 3D volumetric object representation Volumetric Accelerator format (VOLA) is employed. VOLA is a sexaquaternary (power-of-four subdivision) tree-based representation which allows for significant memory saving for volumetric data. Multiple CNN models were trained and pruning techniques for the weights were applied to the trained 3D Volumetric Network in order to remove almost 70% of the parameters and outperform the existing state-of-the-art networks. The top performing and efficient model was ported to the Movidius™Neural Compute Stick (NCS). After deployment on the NCS, it takes 11 ms ( ∼ 90 frames per second) to perform inference on each input volume, with a reported power requirement of 1.2 W, which leads to 75.75 inferences per second per Watt.

VoxSegNet: Volumetric CNNs for Semantic Part Segmentation of 3D Shapes.

Volumetric representation has been widely used for 3D deep learning in shape analysis due to its generalization ability and regular data format. However, for fine-grained tasks like part segmentation, volumetric data has not been widely adopted compared to other representations. Aiming at delivering an effective volumetric method for 3D shape part segmentation, this paper proposes a novel volumetric convolutional neural network. Our method can extract discriminative features encoding detailed information from voxelized 3D data under limited resolution. To this purpose, a spatial dense extraction (SDE) module is designed to preserve spatial resolution during feature extraction procedure, alleviating the loss of details caused by sub-sampling operations such as max pooling. An attention feature aggregation (AFA) module is also introduced to adaptively select informative features from different abstraction levels, leading to segmentation with both semantic consistency and high accuracy of details. Experimental results demonstrate that promising results can be achieved by using volumetric data, with part segmentation accuracy comparable or superior to state-of-the-art non-volumetric methods.

Binary Volumetric Convolutional Neural Networks for 3-D Object Recognition

To address the high computational and memory cost in 3-D volumetric convolutional neural networks (CNNs), we propose an approach to train binary volumetric CNNs for 3-D object recognition. Our method is specifically designed for 3-D data, in which it transforms the inputs and weights in convolutional/fully connected layers to binary values, which can potentially accelerate the networks by efficient bitwise operations. Two loss calculation methods are designed to solve the accuracy decrease problem when the weights in the last layer are binarized. Four binary volumetric CNNs are obtained from their corresponding floating-point networks using our approach. Evaluations on three public datasets from different domains (Computer Aided Design (CAD), light detection and ranging (LiDAR), and RGB-D) show that our binary volumetric CNNs can achieve a comparable recognition performance as their floating-point counterparts but consume less computational and memory resources.

DEEP BV: A FULLY AUTOMATED SYSTEM FOR BRAIN VENTRICLE LOCALIZATION AND SEGMENTATION IN 3D ULTRASOUND IMAGES OF EMBRYONIC MICE.

Volumetric analysis of brain ventricle (BV) structure is a key tool in the study of central nervous system development in embryonic mice. High-frequency ultrasound (HFU) is the only non-invasive, real-time modality available for rapid volumetric imaging of embryos in utero. However, manual segmentation of the BV from HFU volumes is tedious, time-consuming, and requires specialized expertise. In this paper, we propose a novel deep learning based BV segmentation system for whole-body HFU images of mouse embryos. Our fully automated system consists of two modules: localization and segmentation. It first applies a volumetric convolutional neural network on a 3D sliding window over the entire volume to identify a 3D bounding box containing the entire BV. It then employs a fully convolutional network to segment the detected bounding box into BV and background. The system achieves a Dice Similarity Coefficient (DSC) of 0.8956 for BV segmentation on an unseen 111 HFU volume test set surpassing the previous state-of-the-art method (DSC of 0.7119) by a margin of 25%.

Learning Adversarial 3D Model Generation With 2D Image Enhancer

Recent advancements in generative adversarial nets (GANs) and volumetric convolutional neural networks (CNNs) enable generating 3D models from a probabilistic space. In this paper, we have developed a novel GAN-based deep neural network to obtain a better latent space for the generation of 3D models. In the proposed method, an enhancer neural network is introduced to extract information from other corresponding domains (e.g. image) to improve the performance of the 3D model generator, and the discriminative power of the unsupervised shape features learned from the 3D model discriminator. Specifically, we train the 3D generative adversarial networks on 3D volumetric models, and at the same time, the enhancer network learns image features from rendered images. Different from the traditional GAN architecture that uses uninformative random vectors as inputs, we feed the high-level image features learned from the enhancer into the 3D model generator for better training. The evaluations on two large-scale 3D model datasets, ShapeNet and ModelNet, demonstrate that our proposed method can not only generate high-quality 3D models, but also successfully learn discriminative shape representation for classification and retrieval without supervision.

Automatic lesion detection and segmentation of 18F-FET PET in gliomas: A full 3D U-Net convolutional neural network study.

IntroductionAmino-acids positron emission tomography (PET) is increasingly used in the diagnostic workup of patients with gliomas, including differential diagnosis, evaluation of tumor extension, treatment planning and follow-up. Recently, progresses of computer vision and machine learning have been translated for medical imaging. Aim was to demonstrate the feasibility of an automated 18F-fluoro-ethyl-tyrosine (18F-FET) PET lesion detection and segmentation relying on a full 3D U-Net Convolutional Neural Network (CNN).MethodsAll dynamic 18F-FET PET brain image volumes were temporally realigned to the first dynamic acquisition, coregistered and spatially normalized onto the Montreal Neurological Institute template. Ground truth segmentations were obtained using manual delineation and thresholding (1.3 x background). The volumetric CNN was implemented based on a modified Keras implementation of a U-Net library with 3 layers for the encoding and decoding paths. Dice similarity coefficient (DSC) was used as an accuracy measure of segmentation.ResultsThirty-seven patients were included (26 [70%] in the training set and 11 [30%] in the validation set). All 11 lesions were accurately detected with no false positive, resulting in a sensitivity and a specificity for the detection at the tumor level of 100%. After 150 epochs, DSC reached 0.7924 in the training set and 0.7911 in the validation set. After morphological dilatation and fixed thresholding of the predicted U-Net mask a substantial improvement of the DSC to 0.8231 (+ 4.1%) was noted. At the voxel level, this segmentation led to a 0.88 sensitivity [95% CI, 87.1 to, 88.2%] a 0.99 specificity [99.9 to 99.9%], a 0.78 positive predictive value: [76.9 to 78.3%], and a 0.99 negative predictive value [99.9 to 99.9%].ConclusionsWith relatively high performance, it was proposed the first full 3D automated procedure for segmentation of 18F-FET PET brain images of patients with different gliomas using a U-Net CNN architecture.

Toward real-time 3D object recognition: A lightweight volumetric CNN framework using multitask learning

3D data are becoming increasingly popular and easier to access, making 3D information increasingly important for object recognition. Although volumetric convolutional neural networks (CNNs) have been exploited to recognize 3D objects and have achieved notable progress, their computational cost is too high for real-time applications. In this paper, we propose a lightweight volumetric CNN architecture (namely, LightNet) to address the real-time 3D object recognition problem leveraging on multitask learning. We use LightNet to simultaneously predict class and orientation labels from complete and partial shapes. In contrast to the earlier version of this method presented at 3DOR 2017, this extended version introduces batch normalization and better training strategies to improve the recognition accuracy, and also includes more experiments on the newly released large-scale ShapeNet Core55 dataset. Our model has been evaluated on three publicly available benchmarks of complete 3D CAD shapes and incomplete point clouds. Experimental results show that our model achieves the state-of-the-art 3D object recognition performance among shallow volumetric CNNs with the smallest number of training parameters. It is also demonstrated that our method can perform accurate object recognition in real time (less than 6 ms).