3D Residual Network Research Articles

Spatial-Adaptive Siamese Residual Network for Multi-/Hyperspectral Classification

Deep learning methods have been successfully applied for multispectral and hyperspectral images classification due to their ability to extract hierarchical abstract features. However, the performance of these methods relies heavily on large-scale training samples. In this paper, we propose a three-dimensional spatial-adaptive Siamese residual network (3D-SaSiResNet) that requires fewer samples and still enhances the performance. The proposed method consists of two main steps: construction of 3D spatial-adaptive patches and Siamese residual network for multiband images classification. In the first step, the spectral dimension of the original multiband images is reduced by a stacked autoencoder and superpixels of each band are obtained by the simple linear iterative clustering (SLIC) method. Superpixels of the original multiband image can be finally generated by majority voting. Subsequently, the 3D spatial-adaptive patch of each pixel is extracted from the original multiband image by reference to the previously generated superpixels. In the second step, a Siamese network composed of two 3D residual networks is designed to extract discriminative features for classification and we train the 3D-SaSiResNet by pairwise inputting the training samples into the networks. The testing samples are then fed into the trained 3D-SaSiResNet and the learned features of the testing samples are classified by the nearest neighbor classifier. Experimental results on three multiband image datasets show the feasibility of the proposed method in enhancing classification performance even with limited training samples.

A multi-scale 3D convolution neural network for spectral-spatial classification of hyperspectral imagery

Along with the remarkable achievements of various neural networks in the field of computer vision, deep learning methods have gradually been applied to hyperspectral image (HSI) classification. Traditional classification methods have several outstanding issues, including the insufficient hand-craft features and time-consuming and laborious feature extraction. We therefore propose a multi-scale, 3D convolutional neural network (CNN) framework (MSCNN), trained in an end-to-end manner for hyperspectral image classification. Using the original hyperspectral image as an input, the MSCNN framework leverages two branches of a 3D residual neural network to extract deep and abstract HSI features. Then, the HSI spectral-spatial features of different scales are fused and fed into the SoftMax layer to achieve an end-to-end hyperspectral image classification. Besides, data augmentation, dynamic learning rate, and regularization methods are used to promote the rapid convergence of MSCNN and avoid model over-fitting. Three well-known hyperspectral datasets (Indian Pines, University of Pavia and Pavia Center) were used to evaluate the classification performance of the proposed MSCNN method. The results indicate that, compared with the existing deep learning methods, the proposed MSCNN method achieves the best classification performance within the 20 training epochs. The overall accuracy (OA), average accuracy (AA), and kappa statistic (K) of MSCNN are 97.16%, 98.69% and 0.9665, respectively, for Indian Pines; 99.17%, 98.97% and 0.9888 for University of Pavia; and 99.87%, 99.70% and 0.9981 for Pavia Center.

Person Tube Retrieval via Language Description

This paper focuses on the problem of person tube (a sequence of bounding boxes which encloses a person in a video) retrieval using a natural language query. Different from images in person re-identification (re-ID) or person search, besides appearance, person tube contains abundant action and information. We exploit a 2D and a 3D residual networks (ResNets) to extract the appearance and action representation, respectively. To transform tubes and descriptions into a shared latent space where data from the two different modalities can be compared directly, we propose a Multi-Scale Structure Preservation (MSSP) approach. MSSP splits a person tube into several element-tubes on average, whose features are extracted by the two ResNets. Any number of consecutive element-tubes forms a sub-tube. MSSP considers the following constraints for sub-tubes and descriptions in the shared space. 1) Bidirectional ranking. Matching sub-tubes (resp. descriptions) should get ranked higher than incorrect ones for each description (resp. sub-tube). 2) External structure preservation. Sub-tubes (resp. descriptions) from different persons should stay away from each other. 3) Internal structure preservation. Sub-tubes (resp. descriptions) from the same person should be close to each other. Experimental results on person tube retrieval via language description and other two related tasks demonstrate the efficacy of MSSP.

Semi‐supervised convolutional generative adversarial network for hyperspectral image classification

To solve the problem of insufficient annotated samples in hyperspectral image classification, the semi-supervised convolutional generative adversarial network classification model is proposed in this study. The generative adversarial framework constructs an adversarial game, where the generator captures data distribution and generates fake samples, while the discriminator determines whether the input comes from generated or training data. In the proposed method, a deep three-dimensional (3D) convolutional neural network is used to generate the so-called fake cube samples and another 3D deep residual network is designed to discriminate the inputs. Furthermore, the generated samples, labelled and unlabelled samples are put into the discriminator for joint training, and the trained discriminator can determine the authenticity of the sample and the class label. This semi-supervised generative adversarial training strategy can effectively improve the generalisation capability of the deep residual network where the labelled samples are limited. Three widely used hyperspectral images are utilised to evaluate the classification performance of the proposed method: Indian Pines, Pavia University, and Salinas-A. The classification results reveal that the proposed model can improve the classification performance and achieve competitive results compared with the state-of-art methods, especially when there are few training samples.

Single Modality-Based Event Detection Framework for Complex Videos

Event detection of rare and complex events in large video datasets or in unconstrained user-uploaded videos on internet is a challenging task. The presence of irregular camera movement, viewpoint changes, illumination variations and significant changes in the background make extremely difficult to capture underlying motion in videos. In addition, extraction of features using different modalities (single streams) may offer computational complexities and cause abstraction of confusing and irrelevant spatial and semantic features. To address this problem, we present a single stream (RGB only) based on feature of spatial and semantic features extracted by modified 3D Residual Convulsion Network. We combine the spatial and semantic features based on this assumption that difference between both types of features can discover the accurate and relevant features. Moreover, introduction of temporal encoding builds the relationship in consecutive video frames to explore discriminative long-term motion patterns. We conduct extensive experiments on prominent publically available datasets. The obtained results demonstrate the great power of our proposed model and improved accuracy compared with existing state-of-the-art methods.

Learning to Explore Saliency for Stereoscopic Videos via Component-Based Interaction.

In this paper, we devise a saliency prediction model for stereoscopic videos that learns to explore saliency inspired by the component-based interactions including spatial, temporal, as well as depth cues. The model first takes advantage of specific structure of 3D residual network (3D-ResNet) to model the saliency driven by spatio-temporal coherence from consecutive frames. Subsequently, the saliency inferred by implicit-depth is automatically derived based on the displacement correlation between left and right views by leveraging a deep convolutional network (ConvNet). Finally, a component-wise refinement network is devised to produce final saliency maps over time by aggregating saliency distributions obtained from multiple components. In order to further facilitate research towards stereoscopic video saliency, we create a new dataset including 175 stereoscopic video sequences with diverse content, as well as their dense eye fixation annotations. Extensive experiments support that our proposed model can achieve superior performance compared to the state-of-the-art methods on all publicly available eye fixation datasets.

Multiview framework using a 3D residual network for pulmonary micronodule malignancy risk classification.

Pulmonary micronodules account for 80% of all lung nodules. Generally, pulmonary micronodules in the early stages can be detected on thoracic computed tomography (CT) scans. Early diagnosis is crucial for improving the patient's survival rate. This paper aims to estimate the malignancy risk of pulmonary micronodules and potentially improve the survival rate. We extract 3D features of the CT images to obtain richer characteristics. Because superior performance can be achieved by having deep layers, we apply a 3D residual network (3D-ResNet) to classify the pulmonary micronodule. We construct a framework by using three parallel ResNets whose inputs are CT images in different regions of interest, i.e., the multiview of the image. To further evaluate the applicability of the framework, we make a five-category classification and achieve good performance. By fusing different characteristics from three views, we achieve the area under the receiver operating characteristic curve (AUC) of 0.9681. Based on the results of the experiments, our 3D-ResNet has a better performance than 3D-VGG and 3D-Inception in terms of precision (the increase rates are 13.7% and 7.4%), AUC (the increase rates are 15.8% and 5.3%), and accuracy (the increase rates are 14.3% and 4.5%). Meanwhile, the recall performance is close to that of the 3D-Inception network. Overall, the framework we propose has applicability and feasibility in pulmonary micronodule classification.

激光雷达数据协助下的高光谱图像三维残差网络分类

激光雷达数据协助下的高光谱图像三维残差网络分类

A Computer-Aided Detection System for the Detection of Lung Nodules Based on 3D-ResNet

In recent years, the research into automatic aided detection systems for pulmonary nodules has been extremely active. Most of the existing studies are based on 2D convolution neural networks, which cannot make full use of computed tomography’s (CT) 3D spatial information. To address this problem, a computer-aided detection (CAD) system for lung nodules based on a 3D residual network (3D-ResNet) inspired by cognitive science is proposed in this paper. In this system, we feed the slice information extracted from three different axis planes into the U-NET network set, and make the joint decision to generate a candidate nodule set, which is the input of the proposed 3D residual network after extraction. We extracted 3D samples with 40, 44, 48, 52, and 56 mm sides from each candidate nodule in the candidate set and feed them into the trained residual network to get the probability of positive nodule after re-sampling the 3D sample to 48 × 48 × 48 mm 3 . Finally, a joint judgment is made based on the probabilities of five 3D samples of different sizes to obtain the final result. Random rotation and translation and data amplification technology are used to prevent overfitting during network training. The detection intensity on the largest public data set (i.e., the Lung Image Database Consortium and Image Database Resource Initiative—LIDC-IDRI) reached 86.5% and 92.3% at 1 frame per second (FPs) and 4 FPs respectively using our algorithm, which is better than most CAD systems using 2D convolutional neural networks. In addition, a 3D residual network and a multi-section 2D convolution neural network were tested on the unrelated Tianchi dataset. The results indicate that 3D-ResNet has better feature extraction ability than multi-section 2D-ConvNet and is more suitable for reduction of false positive nodules.

3D RANs: 3D Residual Attention Networks for action recognition

In this work, we propose 3D Residual Attention Networks (3D RANs) for action recognition, which can learn spatiotemporal representation from videos. The proposed network consists of attention mechanism and 3D ResNets architecture, and it can capture spatiotemporal information in an end-to-end manner. Specifically, we separately add the attention mechanism along channel and spatial domain to each block of 3D ResNets. For each sliced tensor of an intermediate feature map, we sequentially infer channel and spatial attention maps by channel and spatial attention mechanism submodules in each residual unit block, and the attention maps are multiplied to the input feature map to reweight the key features. We validate our network through extensive experiments in UCF-101, HMDB-51 and Kinetics datasets. Our experiments show that the proposed 3D RANs are superior to the state-of-the-art approaches for action recognition, demonstrating the effectiveness of our networks.

Deep iterative reconstruction estimation (DIRE): approximate iterative reconstruction estimation for low dose CT imaging

The image quality in low dose computed tomography (LDCT) can be severely degraded by amplified mottle noise and streak artifacts. Although the iterative reconstruction (IR) algorithms bring sound improvements, their high computation cost remains a major inconvenient. In this work, a deep iterative reconstruction estimation (DIRE) strategy is developed to estimate IR images from LDCT analytic reconstructions images. Within this DIRE strategy, a 3D residual convolutional network (3D ResNet) architecture is proposed. Experiments on several simulated and real datasets as well as comparisons with state-of-the-art methods demonstrate that the proposed approach is effective in providing improved LDCT images.

A 3D Deep Residual Convolutional Neural Network for Differential Diagnosis of Parkinsonian Syndromes on 18F-FDG PET Images.

Idiopathic Parkinsons disease and atypical parkinsonian syndromes have similar symptoms at early disease stages, which makes the early differential diagnosis difficult. Positron emission tomography with 18F-FDG shows the ability to assess early neuronal dysfunction of neurodegenerative diseases and is well established for clinical use. In the past decades, machine learning methods have been widely used for the differential diagnosis of parkinsonism based on metabolic patterns. Unlike these conventional machine learning methods relying on hand-crafted features, the deep convolutional neural networks, which have achieved significant success in medical applications recently, have the advantage of learning salient feature representations automatically and effectively. This advantage may offer more appropriate invisible features extracted from data for the enhancement of the diagnosis accuracy. Therefore, this paper develops a 3D deep convolutional neural network on 18F-FDG PET images for the automated early diagnosis. Furthermore, we depicted in saliency maps the decision mechanism of the deep learning method to assist the physiological interpretation of deep learning performance. The proposed method was evaluated on a dataset with 920 patients. In addition to improving the accuracy in the differential diagnosis of parkinsonism compared to state-of-the-art approaches, the deep learning methods also discovered saliency features in a number of critical regions (e.g., midbrain), which are widely accepted as characteristic pathological regions for movement disorders but were ignored in the conventional analysis of FDG PET images.

A Deep Learning Model for Predicting Xerostomia Due to Radiation Therapy for Head and Neck Squamous Cell Carcinoma in the RTOG 0522 Clinical Trial

Xerostomia commonly occurs in patients who undergo head and neck radiation therapy and can seriously affect patients' quality of life. In this study, we developed a xerostomia prediction model with radiation treatment data using a 3-dimensional (3D) residual convolutional neural network (rCNN). The model can be used to guide radiation therapy to reduce toxicity. A total of 784 patients with head and neck squamous cell carcinoma enrolled in the Radiation Therapy Oncology Group 0522 clinical trial were included in this study. Late xerostomia is defined as xerostomia of grade ≥2 occurring in the 12th month of radiation therapy. The computed tomography (CT) planning images, 3D dose distributions, and contours of the parotid and submandibular glands were included as 3D rCNN inputs. Comparative experiments were performed for the 3D rCNN model without 1 of the 3 inputs and for the logistic regression model. Accuracy, sensitivity, specificity, F-score, and area under the receiver operator characteristic curve were evaluated. The proposed model achieved promising prediction results. The performance metrics for 3D rCNN model with contour, CT images, and radiation therapy dose; 3D rCNN without contour; 3D rCNN without CT images; 3D rCNN without the dose; logistic regression with the dose and clinical parameters; and logistic regression without clinical parameters were as follows: accuracy: 0.76, 0.74, 0.73, 0.65, 0.64, and 0.56; sensitivity: 0.76, 0.72, 0.77, 0.59, 0.72, and 0.75; specificity: 0.76, 0.76, 0.71, 0.69, 0.59, and 0.43; F-score: 0.70, 0.68, 0.69, 0.56, 0.60, and 0.57; and area under the receiver operator characteristic curve: 0.84, 0.82, 0.78, 0.70, 0.74, and 0.68, respectively. The proposed model uses 3D rCNN filters to extract low- and high-level spatial features and to achieve promising performance. This is a potentially effective model for predicting objective toxicity for supporting clinical decision making.

Domain Progressive 3D Residual Convolution Network to Improve Low-Dose CT Imaging.

The wide applications of X-ray computed tomography (CT) bring low-dose CT (LDCT) into a clinical prerequisite, but reducing the radiation exposure in CT often leads to significantly increased noise and artifacts, which might lower the judgment accuracy of radiologists. In this paper, we put forward a domain progressive 3D residual convolution network (DP-ResNet) for the LDCT imaging procedure that contains three stages: sinogram domain network (SD-net), filtered back projection (FBP), and image domain network (ID-net). Though both are based on the residual network structure, the SD-net and ID-net provide complementary effect on improving the final LDCT quality. The experimental results with both simulated and real projection data show that this domain progressive deep-learning network achieves significantly improved performance by combing the network processing in the two domains.

Analysis of tuberculosis severity levels from CT pulmonary images based on enhanced residual deep learning architecture

This research investigates the application of CT pulmonary images to the detection and characterisation of TB at five levels of severity, in order to monitor the efficacy of treatment. To contend with smaller datasets (i.e. in hundreds) and the characteristics of CT TB images in which abnormalities occupy only limited regions, a 3D block-based residual deep learning network (ResNet) coupled with injection of depth information (depth-ResNet) at each layer was implemented. Progress in evaluation has been accomplished in two ways. One is to assess the proposed depth-ResNet in prediction of severity scores and another is to analyse the probability of high severity of TB. For the former, delivered results are of 92.70 ± 5.97% and 67.15 ± 1.69% for proposed depth-ResNet and ResNet-50 respectively. For the latter, two additional measures are put forward, which are calculated using (1) the overall severity (1 to 5) probability, and (2) separate probabilities of both high severity (scores of 1 to 3) and low severity (scores of 4 and 5) respectively, when scores of 1 to 5 are mapped into initial probabilities of (0.9, 0.7, 0.5, 0.3, 0.2) respectively. As a result, these measures achieve the averaged accuracies of 75.88% and 85.29% for both methods respectively.

An On-Line and Adaptive Method for Detecting Abnormal Events in Videos Using Spatio-Temporal ConvNet

We address in this paper the problem of abnormal event detection in video-surveillance. In this context, we use only normal events as training samples. We propose to use a modified version of pretrained 3D residual convolutional network to extract spatio-temporal features, and we develop a robust classifier based on the selection of vectors of interest. It is able to learn the normal behavior model and detect potentially dangerous abnormal events. This unsupervised method prevents the marginalization of normal events that occur rarely during the training phase since it minimizes redundancy information, and adapt to the appearance of new normal events that occur during the testing phase. Experimental results on challenging datasets show the superiority of the proposed method compared to the state of the art in both frame-level and pixel-level in anomaly detection task.

Toward Automatic Detection of Radiation-Induced Cerebral Microbleeds Using a 3D Deep Residual Network.

Cerebral microbleeds, which are small focal hemorrhages in the brain that are prevalent in many diseases, are gaining increasing attention due to their potential as surrogate markers of disease burden, clinical outcomes, and delayed effects of therapy. Manual detection is laborious and automatic detection and labeling of these lesions is challenging using traditional algorithms. Inspired by recent successes of deep convolutional neural networks in computer vision, we developed a 3D deep residual network that can distinguish true microbleeds from false positive mimics of a previously developed technique based on traditional algorithms. A dataset of 73 patients with radiation-induced cerebral microbleeds scanned at 7T with susceptibility-weighted imaging was used to train and evaluate our model. With the resulting network, we maintained 95% of the true microbleeds in 12 test patients and the average number of false positives was reduced by 89%, achieving a detection precision of 71.9%, higher than existing published methods. The likelihood score predicted by the network was also evaluated by comparing to a neuroradiologist's rating, and good correlation was observed.