Pre-trained Deep Convolutional Neural Network Research Articles

Encoding Pose Features to Images With Data Augmentation for 3-D Action Recognition

Recently, numerous methods have been introduced for three-dimensional (3-D) action recognition using handcrafted feature descriptors coupled traditional classifiers. However, they cannot learn high-level features of a whole skeleton sequence exhaustively. In this paper, a novel encoding technique - namely, pose feature to image (PoF2I), is introduced to transform the pose features of joint-joint distance and orientation to color pixels. By concatenating the features of all skeleton frames in a sequence, a color image is generated to depict spatial joint correlations and temporal pose dynamics of an action appearance. The strategy of end-to-end fine-tuning a pretrained deep convolutional neural network, which completely capture multiple high-level features at multiscale action representation, is implemented for learning recognition models. We further propose an efficient data augmentation mechanism for informative enrichment and overfitting prevention. The experimental results on six challenging 3-D action recognition datasets demonstrate that the proposed method outperforms state-of-the-art approaches.

Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures

AbstractHealth monitoring of civil infrastructures is a key application of Internet of things (IoT), while edge computing is an important component of IoT. In this context, swarms of autonomous inspection robots, which can replace current manual inspections, are examples of edge devices. Incorporation of pretrained deep learning algorithms into these robots for autonomous damage detection is a challenging problem since these devices are typically limited in computing and memory resources. This study introduces a solution based on network pruning using Taylor expansion to utilize pretrained deep convolutional neural networks for efficient edge computing and incorporation into inspection robots. Results from comprehensive experiments on two pretrained networks (i.e., VGG16 and ResNet18) and two types of prevalent surface defects (i.e., crack and corrosion) are presented and discussed in detail with respect to performance, memory demands, and the inference time for damage detection. It is shown that the proposed approach significantly enhances resource efficiency without decreasing damage detection performance.

Brain tumor classification for MR images using transfer learning and fine-tuning

Accurate and precise brain tumor MR images classification plays important role in clinical diagnosis and decision making for patient treatment. The key challenge in MR images classification is the semantic gap between the low-level visual information captured by the MRI machine and the high-level information perceived by the human evaluator. The traditional machine learning techniques for classification focus only on low-level or high-level features, use some handcrafted features to reduce this gap and require good feature extraction and classification methods. Recent development on deep learning has shown great progress and deep convolution neural networks (CNNs) have succeeded in the images classification task. Deep learning is very powerful for feature representation that can depict low-level and high-level information completely and embed the phase of feature extraction and classification into self-learning but require large training dataset in general. For most of the medical imaging scenario, the training datasets are small, therefore, it is a challenging task to apply the deep learning and train CNN from scratch on the small dataset. Aiming this problem, we use pre-trained deep CNN model and propose a block-wise fine-tuning strategy based on transfer learning. The proposed method is evaluated on T1-weighted contrast-enhanced magnetic resonance images (CE-MRI) benchmark dataset. Our method is more generic as it does not use any handcrafted features, requires minimal preprocessing and can achieve average accuracy of 94.82% under five-fold cross-validation. We compare our results not only with the traditional machine learning but also with deep learning methods using CNNs. Experimental results show that our proposed method outperforms state-of-the-art classification on the CE-MRI dataset.

Assessment of Critical Feeding Tube Malpositions on Radiographs Using Deep Learning

Assess the efficacy of deep convolutional neural networks (DCNNs) in detection of critical enteric feeding tube malpositions on radiographs. 5475 de-identified HIPAA compliant frontal view chest and abdominal radiographs were obtained, consisting of 174 x-rays of bronchial insertions and 5301 non-critical radiographs, including normal course, normal chest, and normal abdominal x-rays. The ground-truth classification for enteric feeding tube placement was performed by two board-certified radiologists. Untrained and pretrained deep convolutional neural network models for Inception V3, ResNet50, and DenseNet 121 were each employed. The radiographs were fed into each deep convolutional neural network, which included untrained and pretrained models. The Tensorflow framework was used for Inception V3, ResNet50, and DenseNet. Images were split into training (4745), validation (630), and test (100). Both real-time and preprocessing image augmentation strategies were performed. Receiver operating characteristic (ROC) and area under the curve (AUC) on the test data were used to assess the models. Statistical differences among the AUCs were obtained. p < 0.05 was considered statistically significant. The pretrained Inception V3, which had an AUC of 0.87 (95 CI; 0.80–0.94), performed statistically significantly better (p < .001) than the untrained Inception V3, with an AUC of 0.60 (95 CI; 0.52–0.68). The pretrained Inception V3 also had the highest AUC overall, as compared with ResNet50 and DenseNet121, with AUC values ranging from 0.82 to 0.85. Each pretrained network outperformed its untrained counterpart. (p < 0.05). Deep learning demonstrates promise in differentiating critical vs. non-critical placement with an AUC of 0.87. Pretrained networks outperformed untrained ones in all cases. DCNNs may allow for more rapid identification and communication of critical feeding tube malpositions.

Identifying Images of Invasive Hydrangea Using Pre-Trained Deep Convolutional Neural Networks

Invasive species are threatening habitats of native species in many countries around the world. The current methods of monitoring them depend on expert knowledge. Trained scientists visit designated areas and take note of the species inhabiting them. Using such a highly qualified workforce is expensive, time inefficient and insufficient since humans cannot cover large areas when sampling. In this paper, machine learning based approach is presented for identifying images of invasive hydrangea (a beautiful invasive species original of Asia) with a dataset that contains approximately 3,800 images taken in a Brazilian national forest and in some of the pictures there is Hydrangea. A deep learning technique that extensively applied to image recognition was used. Our trained model achieved an accuracy of 99.71% on a held-out test set, demonstrating the feasibility of this approach.

Transfer learning and feature fusion for kinship verification

Facial image analysis has been an important subject of study in the communities of pattern recognition and computer vision. Facial images contain much information about the person they belong to like identity, age, gender, ethnicity, and expression. This paper introduces a new framework that exploits facial deep features for kinship verification. The framework integrates efficient feature selection and kinship-oriented discriminant data projection. The resulting framework incorporates three levels of fusion: (1) an early fusion of descriptors where the filter selection selects the most relevant deep features, (2) a middle-stage fusion which exploits a kinship-based multi-view metric learning method, and (3) a late-stage fusion that merges classifiers responses. In our study, face features are provided by the pre-trained deep convolutional neural networks VGG-F and VGG-Face that were originally proposed for discriminating categories of objects and identities, respectively. Experimental results on two benchmarked datasets for kinship verification in the wild (KinFaceW-I and KinFaceW-II) show that the proposed framework outperforms state-of-the-art techniques without the use of external data or data augmentation that are tailored for the kinship verification problem. These experiments show that the proposed scheme can outperform feature fusion obtained by deep multi-metric learning.

Person classification from aerial imagery using local convolutional neural network features

ABSTRACTThe need for search and rescue is not one that will go away. Extending beyond the concerns of hikers, natural disasters have brought the necessity for search and rescue in the heart of civilization. The advent of new technologies, such as unmanned aerial vehicles, can help decrease the cost of search and rescue operations and increase survival rates by finding the lost individuals in a faster manner. Visual object recognition is key to successful drone application for assisting search and rescue activities, and it is critical to develop a fully autonomous system. In recent years, deep convolutional neural networks have proven themselves as a powerful class of models and have become de facto a standard in a computer vision community. Moreover, a number of studies have shown that intermediate activations extracted from a deep convolutional neural network, pre-trained on large image dataset, can be adopted as universal image representation and transferred to other image classification tasks, leading to striking performances. In this paper, we investigate the effectiveness of different layer activations on the performance of convolutional features for binary person classification from UAV imagery, using various deep network architectures. Experiments have demonstrated that convolutional network generated features can deliver extremely competitive results, comparable to human-level performance. Furthermore, they reveal that in said context fully connected layer activations generalize well and are more expressive than the mid- or higher-level layers activations. Our work provides guidance for transferring pre-trained deep convolutional neural networks to address remote person classification tasks.

A deep learning- and partial least square regression-based model observer for a low-contrast lesion detection task in CT.

This work aims to develop a new framework of image quality assessment using deep learning-based model observer (DL-MO) and to validate it in a low-contrast lesion detection task that involves CT images with patient anatomical background. The DL-MO was developed using the transfer learning strategy to incorporate a pretrained deep convolutional neural network (CNN), a partial least square regression discriminant analysis (PLS-DA) model and an internal noise component. The CNN was previously trained to achieve the state-of-the-art classification accuracy over a natural image database. The earlier layers of the CNN were used as a deep feature extractor, with the assumption that similarity exists between the CNN and the human visual system. The PLSR model was used to further engineer the deep feature for the lesion detection task in CT images. The internal noise component was incorporated to model the inefficiency and variability of human observer (HO) performance, and to generate the ultimate DL-MO test statistics. Seven abdominal CT exams were retrospectively collected from the same type of CT scanners. To compare DL-MO with HO, 12 experimental conditions with varying lesion size, lesion contrast, radiation dose, and reconstruction types were generated, each condition with 154 trials. CT images of a real liver metastatic lesion were numerically modified to generate lesion models with four lesion sizes (5, 7, 9, and 11mm) and three contrast levels (15, 20, and 25HU). The lesions were inserted into patient liver images using a projection-based method. A validated noise insertion tool was used to synthesize CT exams with 50% and 25% of routine radiation dose level. CT images were reconstructed using the weighted filtered back projection algorithm and an iterative reconstruction algorithm. Four medical physicists performed a two-alternative forced choice (2AFC) detection task (with multislice scrolling viewing mode) on patient images across the 12 experimental conditions. DL-MO was operated on the same datasets. Statistical analyses were performed to evaluate the correlation and agreement between DL-MO and HO. A statistically significant positive correlation was observed between DL-MO and HO for the 2AFC low-contrast detection task that involves patient liver background. The corresponding Pearson product moment correlation coefficient was 0.986 [95% confidence interval (0.950, 0.996)]. Bland-Altman agreement analysis did not indicate statistically significant differences. The proposed DL-MO is highly correlated with HO in a low-contrast detection task that involves realistic patient liver background. This study demonstrated the potential of the proposed DL-MO to assess image quality directly based on patient images in realistic, clinically relevant CT tasks.

Transfer Learning with Deep Convolutional Neural Networks for Classifying Cellular Morphological Changes

The quantification and identification of cellular phenotypes from high-content microscopy images has proven to be very useful for understanding biological activity in response to different drug treatments. The traditional approach has been to use classical image analysis to quantify changes in cell morphology, which requires several nontrivial and independent analysis steps. Recently, convolutional neural networks have emerged as a compelling alternative, offering good predictive performance and the possibility to replace traditional workflows with a single network architecture. In this study, we applied the pretrained deep convolutional neural networks ResNet50, InceptionV3, and InceptionResnetV2 to predict cell mechanisms of action in response to chemical perturbations for two cell profiling datasets from the Broad Bioimage Benchmark Collection. These networks were pretrained on ImageNet, enabling much quicker model training. We obtain higher predictive accuracy than previously reported, between 95% and 97%. The ability to quickly and accurately distinguish between different cell morphologies from a scarce amount of labeled data illustrates the combined benefit of transfer learning and deep convolutional neural networks for interrogating cell-based images.

A Deep-Learning-Based Geological Parameterization for History Matching Complex Models

A new low-dimensional parameterization based on principal component analysis (PCA) and convolutional neural networks (CNN) is developed to represent complex geological models. The CNN-PCA method is inspired by recent developments in computer vision using deep learning. CNN-PCA can be viewed as a generalization of an existing optimization-based PCA (O-PCA) method. Both CNN-PCA and O-PCA entail post-processing a PCA model to better honor complex geological features. In CNN-PCA, rather than use a histogram-based regularization as in O-PCA, a new regularization involving a set of metrics for multipoint statistics is introduced. The metrics are based on summary statistics of the nonlinear filter responses of geological models to a pre-trained deep CNN. In addition, in the CNN-PCA formulation presented here, a convolutional neural network is trained as an explicit transform function that can post-process PCA models quickly. CNN-PCA is shown to provide both unconditional and conditional realizations that honor the geological features present in reference SGeMS geostatistical realizations for a binary channelized system. Flow statistics obtained through simulation of random CNN-PCA models closely match results for random SGeMS models for a demanding case in which O-PCA models lead to significant discrepancies. Results for history matching are also presented. In this assessment CNN-PCA is applied with derivative-free optimization, and a subspace randomized maximum likelihood method is used to provide multiple posterior models. Data assimilation and significant uncertainty reduction are achieved for existing wells, and physically reasonable predictions are also obtained for new wells. Finally, the CNN-PCA method is extended to a more complex non-stationary bimodal deltaic fan system, and is shown to provide high-quality realizations for this challenging example.

Automatic fabric defect detection using a deep convolutional neural network

Fabric defect detection plays an important role in the textile production process, but there are still some challenges in detecting defects rapidly and accurately. In this paper, we propose a powerful detection method for automatic fabric defect detection using a deep convolutional neural network (CNN). It consists of three main steps. First, the fabric image is decomposed into local patches and each local patch is labelled. Then the labelled patches are transmitted to the pretrained deep CNN for transfer learning. Finally, defects are detected during the inspection phase by sliding over the whole image using the trained model, and the category and position of each defect is obtained. The proposed method is validated on two public and one self‐made fabric database. The experimental results demonstrate that our method significantly outperforms selected state‐of‐the art methods in terms of both quality and robustness.

Efficient deep features selections and classification for flower species recognition

Image-based automatic flower species classification is an important problem for the biologists who construct digital flower catalogs. A dozen of work about flower species recognition has been proposed so far based on traditional image processing routines. Nowadays, researchers apply the deep learning on various image-based object recognition tasks. In this paper, deep convolutional neural networks (DCNN) based hybrid method is applied to flower species classification. The proposed method initially employs a pre-trained DCNN model for feature extraction. To this end, two popular DCNN architectures namely, AlexNet and VGG16 models are adopted. For constructing efficient feature sets, the features from AlexNet and VGG16 models are then concatenated. Finally, a feature selection algorithm, the minimum Redundancy Maximum Relevance (mRMR) method, is used to select the more efficient features. A support vector machine (SVM) classifier with Radial Bases Function (RBF) kernel is employed to classify the flower species using the extracted features. Flower17 and Flower102 datasets which have a huge amount of category are used in the experimental works. Various experiments results show that we have achieved 96.39% and 95.70% accuracy performance for Flower17 and Flower102, respectively. The obtained results demonstrate the effectiveness of the proposed method, despite the relative simplicity of the approach.

Semantic segmentation of multisensor remote sensing imagery with deep ConvNets and higher-order conditional random fields

Aerial images acquired by multiple sensors provide comprehensive and diverse information of materials and objects within a surveyed area. The current use of pretrained deep convolutional neural networks (DCNNs) is usually constrained to three-band images (i.e., RGB) obtained from a single optical sensor. Additional spectral bands from a multiple sensor setup introduce challenges for the use of DCNN. We fuse the RGB feature information obtained from a deep learning framework with light detection and ranging (LiDAR) features to obtain semantic labeling. Specifically, we propose a decision-level multisensor fusion technique for semantic labeling of the very-high-resolution optical imagery and LiDAR data. Our approach first obtains initial probabilistic predictions from two different sources: one from a pretrained neural network fine-tuned on a three-band optical image, and another from a probabilistic classifier trained on LiDAR data. These two predictions are then combined as the unary potential using a higher-order conditional random field (CRF) framework, which resolves fusion ambiguities by exploiting the spatial–contextual information. We utilize graph cut to efficiently infer the final semantic labeling for our proposed higher-order CRF framework. Experiments performed on three benchmarking multisensor datasets demonstrate the performance advantages of our proposed method.

Transferring Ensemble Representations Using Deep Convolutional Neural Networks for Small-Scale Image Classification

The deep convolutional neural networks (DCNN) require large number of training data to avoid overfitting, which makes it unsuitable for processing small-scale image datasets. The transfer learning using DCNN (TCNN) reuses pre-trained layers to generate a mid-level image representation so that the optimization of more than millions CNN parameters can be avoided. By this way, overfitting problem in small-scale data can be alleviated. However, although now many public DCNNs have been trained and can be reused, the existing TCNNs are formed by only a single pre-trained DCNN structure and cannot make full use of multiple structures of pre-trained DCNNs. At the same time, the existing ensemble CNNs have not enough good representation ability. To address this problem, we combine the conventional ideas of ensemble CNNs and propose three ensemble TCNNs (TECNN). They are the voting method based on the combination of all TCNNs, the PickOver method by finding the optimal combination, and weighted method by finding weighted combination. Different from the existing ensemble CNNs, the proposed methods do not need to retrain the component CNNs and generate ensemble transferring representations by transferring the pre-trained mid-level parameters. The mathematical models of those three methods are also provided. Their versions of using fine-tuning are also compared in the experiments. In addition, we replace the Softmax classifier with ensemble linear classifiers in the full-connection layer. They outperform the current state of the art algorithms on Caltech ImageNet and some internet image data. All this research has released as an open source library called Transferring Image Ensemble Representations using Deep Convolutional Neural Networks (TECNN). The source codes and relevant datasets in different versions are available from: http://www.cquptshuyinxia.com/TECNN.html .

Feasibility of Image Registration for Ultrasound-Guided Prostate Radiotherapy Based on Similarity Measurement by a Convolutional Neural Network.

Purpose:Registration of 3-dimensional ultrasound images poses a challenge for ultrasound-guided radiation therapy of the prostate since ultrasound image content changes significantly with anatomic motion and ultrasound probe position. The purpose of this work is to investigate the feasibility of using a pretrained deep convolutional neural network for similarity measurement in image registration of 3-dimensional transperineal ultrasound prostate images.Methods:We propose convolutional neural network-based registration that maximizes a similarity score between 2 identical in size 3-dimensional regions of interest: one encompassing the prostate within a simulation (reference) 3-dimensional ultrasound image and another that sweeps different spatial locations around the expected prostate position within a pretreatment 3-dimensional ultrasound image. The similarity score is calculated by (1) extracting pairs of corresponding 2-dimensional slices (patches) from the regions of interest, (2) providing these pairs as an input to a pretrained convolutional neural network which assigns a similarity score to each pair, and (3) calculating an overall similarity by summing all pairwise scores. The convolutional neural network method was evaluated against ground truth registrations determined by matching implanted fiducial markers visualized in a pretreatment orthogonal pair of x-ray images. The convolutional neural network method was further compared to manual registration and a standard commonly used intensity-based automatic registration approach based on advanced normalized correlation.Results:For 83 image pairs from 5 patients, convolutional neural network registration errors were smaller than 5 mm in 81% of the cases. In comparison, manual registration errors were smaller than 5 mm in 61% of the cases and advanced normalized correlation registration errors were smaller than 5 mm only in 25% of the cases.Conclusion:Convolutional neural network evaluation against manual registration and an advanced normalized correlation -based registration demonstrated better accuracy and reliability of the convolutional neural network. This suggests that with training on a large data set of transperineal ultrasound prostate images, the convolutional neural network method has potential for robust ultrasound-to-ultrasound registration.

Classification of UAV-to-Ground Vehicles Based on Micro-Doppler Signatures Using Singular Value Decomposition and Deep Convolutional Neural Networks

Attack from the unmanned aerial vehicles (UAVs) has been the main means of high-precision strike. Therefore, classifying ground vehicles from the UAV with high accuracy is of great significance. In order to avoid the complex feature extracting process and realize the classification of UAV-to-ground vehicles in different situations, this paper proposed a method based on micro-Doppler signatures using singular value decomposition (SVD) and deep convolutional neural networks (DCNNs). First, models of UAV-to-ground vehicles are built to analyze the micro-Doppler components and Doppler signals in five different cases are given. Second, time-frequency spectrums of Doppler signals with low signal-to-noise ratios are improved after removing noise using SVD. Third, transfer-learning of pre-trained DCNNs is achieved using measured data and classification under various conditions is realized using the new-trained network. When there is no noise, the overall classification accuracy of two types of Doppler signals, three types of Doppler signals, four types of Doppler signals and five types of Doppler signals has reached 100%, 97%, 97%, and 96%, respectively. Comparison with current methods which need to extract micro-Doppler features by time-frequency techniques are also made. Outstanding performance proves the superiority and robustness of the proposed method.

Dictionaries of deep features for land-use scene classification of very high spatial resolution images

Land-use classification in very high spatial resolution images is critical in the remote sensing field. Consequently, remarkable efforts have been conducted towards developing increasingly accurate approaches for this task. In recent years, deep learning has emerged as a dominant paradigm for machine learning, and methodologies based on deep convolutional neural networks have received particular attention from the remote sensing community. These methods typically utilize transfer learning and/or data augmentation to accommodate a small number of labeled images in the publicly available datasets in this field. However, they typically require powerful computers and/or a long time for training. In this work, we propose a simple and novel method for land-use classification in very high spatial resolution images, which efficiently combines transfer learning with a sparse representation. Specifically, the proposed method performs the classification of land-use scenes using a modified version of the well-known sparse representation-based classification method. While this method directly uses the training images to form dictionaries, which are employed to classify test images, our method utilizes a pre-trained deep convolutional neural network and the Gaussian mixture model to generate more robust and compact “dictionaries of deep features.” The effectiveness of the proposed method was evaluated on two publicly available datasets: UC Merced and Brazilian Cerrado–Savana. The experimental results suggest that our method can potentially outperform state-of-the-art techniques for land-use classification in very high spatial resolution images.

Affective image classification by jointly using interpretable art features and semantic annotations

Affective image classification, which aims to classify images according to their affective characteristics of inducing human emotions, has drawn increasing research attentions in the multimedia community. Although many features have been attempted, the semantic gap between low-level visual features and high-level emotional semantics, however, remains a major challenge. In this paper, we propose an affective image classification algorithm by jointly using the visual features extracted under the guidance of the art theory and semantic image annotations, such as the categories of objects and scenes, generated by a pre-trained deep convolutional neural network. This algorithm has been evaluated against three state-of-the-art approaches on three benchmark image datasets. Our results indicate that combining interpretable aesthetic features and semantic annotations can better characterize the emotional semantics and the proposed algorithm is able to produce more accurate affective image classification than the other three approaches.

The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment.

Recently, researchers have built new deep learning (DL) models using a single image modality to diagnose age-related macular degeneration (AMD). Retinal fundus and optical coherence tomography (OCT) images in clinical settings are the most important modalities investigating AMD. Whether concomitant use of fundus and OCT data in DL technique is beneficial has not been so clearly identified. This experimental analysis used OCT and fundus image data of postmortems from the Project Macula. The DL based on OCT, fundus, and combination of OCT and fundus were invented to diagnose AMD. These models consisted of pre-trained VGG-19 and transfer learning using random forest. Following the data augmentation and training process, the DL using OCT alone showed diagnostic efficiency with area under the curve (AUC) of 0.906 (95% confidence interval, 0.891-0.921) and 82.6% (81.0-84.3%) accuracy rate. The DL using fundus alone exhibited AUC of 0.914 (0.900-0.928) and 83.5% (81.8-85.0%) accuracy rate. Combined usage of the fundus with OCT increased the diagnostic power with AUC of 0.969 (0.956-0.979) and 90.5% (89.2-91.8%) accuracy rate. The Delong test showed that the DL using both OCT and fundus data outperformed the DL using OCT alone (P value < 0.001) and fundus image alone (P value < 0.001). This multimodal random forest model showed even better performance than a restricted Boltzmann machine (P value = 0.002) and deep belief network algorithms (P value = 0.042). According to Duncan's multiple range test, the multimodal methods significantly improved the performance obtained by the single-modal methods. In this preliminary study, a multimodal DL algorithm based on the combination of OCT and fundus image raised the diagnostic accuracy compared to this data alone. Future diagnostic DL needs to adopt the multimodal process to combine various types of imaging for a more precise AMD diagnosis. Graphical abstract The basic architectural structure of the tested multimodal deep learning model based on pre-trained deep convolutional neural network and random forest using the combination of OCT and fundus image.

Age estimation in facial images through transfer learning

This paper was aimed to address the problem of image-based human age estimation. It has the following main contributions. First, we provide a comparison of three hand-crafted image features and five deep convolutional neural networks (DCNNs). Secondly, we show that the use of pre-trained DCNNs as feature extractors can transfer the knowledge of DCNNs to new datasets and domains that were not necessarily addressed in the training phase. This is achieved by only retraining a shallow regressor over the deep features. Thirdly, we provide a cross-database evaluation involving biological and apparent ages. The paper shows that transfer learning allows the use of pre-trained DCNNs regardless of the type of ages (apparent or biological) that is adopted in DCNN training. The experiments are carried out on three public databases: MORPH, PAL, and Chalearn2016.