Pre-trained 2D Convolutional Neural Network Research Articles

Bridge Damage Identification Using Deep Neural Networks on Time-Frequency Signals Representation.

For the purpose of maintaining and prolonging the service life of civil constructions, structural damage must be closely monitored. Monitoring the incidence, formation, and spread of damage is crucial to ensure a structure's ongoing performance. This research proposes a unique approach for multiclass damage detection using acceleration responses based on synchrosqueezing transform (SST) together with deep learning algorithms. In particular, our pipeline is able to classify correctly the time series representing the responses of accelerometers placed on a bridge, which are classified with respect to different types of damage scenarios applied to the bridge. Using benchmark data from the Z24 bridge for multiclass classification for different damage situations, the suggested method is validated. This dataset includes labeled accelerometer measurements from a real-world bridge that has been gradually damaged by various conditions. The findings demonstrate that the suggested approach is successful in exploiting pre-trained 2D convolutional neural networks, obtaining a high classification accuracy that can be further boosted by the application of simple voting methods.

2.75D: Boosting learning by representing 3D Medical imaging to 2D features for small data

In medical-data driven learning, 3D convolutional neural networks (CNNs) have started to show superior performance to 2D CNNs in numerous deep learning tasks, proving the added value of 3D spatial information in feature representation. However, the difficulty in collecting more training samples to converge, more computational resources and longer execution time make this approach less applied. Also, applying transfer learning on 3D CNN is challenging due to a lack of publicly available pre-trained 3D models. To tackle these issues, we proposed a novel 2D strategical representation of volumetric data, namely 2.75D. In this work, the spatial information of 3D images is captured in a single 2D view by a spiral-spinning technique. As a result, 2D CNN networks can also be used to learn volumetric information. Besides, we can fully leverage pre-trained 2D CNNs for downstream vision problems. We also explore a multi-view 2.75D strategy, 2.75D 3 channels (2.75D × 3), to boost the advantage of 2.75D. We evaluated the proposed methods on three public datasets with different modalities or organs (Lung CT, Breast MRI, and Prostate MRI), against their 2D, 2.5D, and 3D counterparts in classification tasks. Results show that the proposed methods significantly outperform other counterparts when all methods were trained from scratch on the lung dataset. Such performance gain is more pronounced with transfer learning or in the case of limited training data. Our methods also achieved comparable performance on other datasets. In addition, our methods achieved a substantial reduction in time consumption of training and inference compared with the 2.5D or 3D method.

A Class Activation Map-Based Interpretable Transfer Learning Model for Automated Detection of ADHD from fMRI Data.

Automatic detection of Attention Deficit Hyperactivity Disorder (ADHD) based on the functional Magnetic Resonance Imaging (fMRI) through Deep Learning (DL) is becoming a quite useful methodology due to the curse of-dimensionality problem of the data is solved. Also, this method proposes an invasive and robust solution to the variances in data acquisition and class distribution imbalances. In this paper, a transfer learning approach, specifically ResNet-50 type pre-trained 2D-Convolutional Neural Network (CNN) was used to automatically classify ADHD and healthy children. The results demonstrated that ResNet-50 architecture with 10-k cross-validation (CV) achieves an overall classification accuracy of 93.45%. The interpretation of the results was done via the Class Activation Map (CAM) analysis which showed that children with ADHD differed from controls in a wide range of brain areas including frontal, parietal and temporal lobes.

A 3D CNN APPROACH FOR CHANGE DETECTION IN HR SATELLITE IMAGE TIME SERIES BASED ON A PRETRAINED 2D CNN

Abstract. Over recent decades, Change Detection (CD) has been intensively investigated due to the availability of High Resolution (HR) multi-spectral multi-temporal remote sensing images. Deep Learning (DL) based methods such as Convolutional Neural Network (CNN) have recently received increasing attention in CD problems demonstrating high potential. However, most of the CNN-based CD methods are designed for bi-temporal image analysis. Here, we propose a Three-Dimensional (3D) CNN-based CD approach that can effectively deal with HR image time series and process spatial-spectral-temporal features. The method is unsupervised and thus does not require the complex task of collecting labelled multi-temporal data. Since there are only a few pretrained 3D CNNs available that are not suitable for remote sensing CD analysis, the proposed approach starts with a pretrained 2D CNN architecture trained on remote sensing images for semantic segmentation and develops a 3D CNN architecture using a transfer learning technique to jointly deal with spatial, spectral and temporal information. A layerwise feature reduction strategy is performed to select the most informative features and a pixelwise year-based Change Vector Analysis (CVA) is employed to identify changed pixels. Experimental results on a long time series of Landsat 8 images for an area located in Saudi Arabia confirm the effectiveness of the proposed approach.

Incorporating Radiomics into Machine Learning Models to Predict Outcomes of Neuroblastoma.

Neuroblastoma is one of the most common pediatric cancers. This study used machine learning (ML) to predict the mortality and a few other investigated intermediate outcomes of neuroblastoma patients non-invasively from CT images. Performances of multiple ML algorithms over retrospective CT images of 65 neuroblastoma patients are analyzed. An artificial neural network (ANN) is used on tumor radiomic features extracted from 3D CT images. A pre-trained 2D convolutional neural network (CNN) is used on slices of the same images. ML models are trained for various pathologically investigated outcomes of these patients. A subspecialty-trained pediatric radiologist independently reviewed the manually segmented primary tumors. Pyradiomics library is used to extract 105 radiomic features. Six ML algorithms are compared to predict the following outcomes: mortality, presence or absence of metastases, neuroblastoma differentiation, mitosis-karyorrhexis index (MKI), presence or absence of MYCN gene amplification, and presence of image-defined risk factors (IDRF). The prediction ranges over multiple experiments are measured using the area under the receiver operating characteristic (ROC-AUC) for comparison. Our results show that the radiomics-based ANN method slightly outperforms the other algorithms in predicting all outcomes except classification of the grade of neuroblastic differentiation, for which the elastic regression model performed the best. Contributions of the article are twofold: (1) noninvasive models for the prognosis from CT images of neuroblastoma, and (2) comparison of relevant ML models on this medical imaging problem.

Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks

Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.

Structural and diffusion MRI based schizophrenia classification using 2D pretrained and 3D naive Convolutional Neural Networks

The ability of automatic feature learning makes Convolutional Neural Network (CNN) potentially suitable to uncover the complex and widespread brain changes in schizophrenia. Despite that, limited studies have been done on schizophrenia identification using interpretable deep learning approaches on multimodal neuroimaging data. Here, we developed a deep feature approach based on pre-trained 2D CNN and naive 3D CNN models trained from scratch for schizophrenia classification by integrating 3D structural and diffusion magnetic resonance imaging (MRI) data. We found that the naive 3D CNN models outperformed the pretrained 2D CNN models and the handcrafted feature-based machine learning approach using support vector machine during both cross-validation and testing on an independent dataset. Multimodal neuroimaging-based models accomplished performance superior to models based on a single modality. Furthermore, we identified brain grey matter and white matter regions critical for illness classification at the individual- and group-level which supported the salience network and striatal dysfunction hypotheses in schizophrenia. Our findings underscore the potential of CNN not only to automatically uncover and integrate multimodal 3D brain imaging features for schizophrenia identification, but also to provide relevant neurobiological interpretations which are crucial for developing objective and interpretable imaging-based probes for prognosis and diagnosis in psychiatric disorders.

Utilizing transfer learning of pre-trained AlexNet and relevance vector machine for regression for predicting healthy older adult’s brain age from structural MRI

Discrepancies between the estimated brain age from brain structural MRI and the chronological age have been associated with a broad spectrum of neurocognitive disorders. The performance of brain age estimation heavily depends on predefined or hand-crafted features. Although 3D convolutional neural network (CNN) based approaches have been proposed, they require high computational cost, large memory load, and numerous images. Coupling a pre-trained 2D CNN for transfer learning with a well-established relevance vector machine for regression approach can greatly enhance the capabilities of the model. Several important strategies, including feature transfer learning, 3D feature concatenation, and dimensionality reduction were taken. The estimated brain age was modeled by structural magnetic resonance imaging (sMRI) from 594 normal healthy older individuals (age 50–90 years). We proposed and manifested a pre-trained AlexNet as a robust feature extractor. Also, the considerable cost of developing a 3D CNN was avoided by applying 3D feature concatenation and data reduction. The proposed method achieves superior performance with a mean absolute error of 4.51 years for old subjects. The predicted brain age also demonstrated high test-retest reliability (intra-class correlation coefficient of 0.979). The effectiveness and robustness of the proposed model were well studied. The proposed approach can compete with or even outperform those state-of-the-art approaches, and feature transfer learning strategy can introduce new perspectives to some well-established brain age prediction models with predefined or hand-crafted features.

3D Convolutional Neural Networks Initialized from Pretrained 2D Convolutional Neural Networks for Classification of Industrial Parts.

Deep learning methods have been successfully applied to image processing, mainly using 2D vision sensors. Recently, the rise of depth cameras and other similar 3D sensors has opened the field for new perception techniques. Nevertheless, 3D convolutional neural networks perform slightly worse than other 3D deep learning methods, and even worse than their 2D version. In this paper, we propose to improve 3D deep learning results by transferring the pretrained weights learned in 2D networks to their corresponding 3D version. Using an industrial object recognition context, we have analyzed different combinations of 3D convolutional networks (VGG16, ResNet, Inception ResNet, and EfficientNet), comparing the recognition accuracy. The highest accuracy is obtained with EfficientNetB0 using extrusion with an accuracy of 0.9217, which gives comparable results to state-of-the art methods. We also observed that the transfer approach enabled to improve the accuracy of the Inception ResNet 3D version up to 18% with respect to the score of the 3D approach alone.

An automated and efficient convolutional architecture for disguise-invariant face recognition using noise-based data augmentation and deep transfer learning

Face recognition is diversely used in modern biometric and security applications. Most of the current face recognition techniques show good results in a constrained environment. However, these techniques face many problems in real-world scenarios such as low-quality images, temporal variations and facial disguises creating variations in facial features. The reason for these deteriorating results is the employment of handcrafted features having weak generalization capabilities and neglecting the complexities associated with domain adaption in case of deep learning models. In this paper, we have studied the efficacy of deep learning methods incorporating simple noise-based data augmentation for disguise invariant face recognition (DIFR). The proposed method detects face in an image using Viola Jones face detector and classifies it using a pre-trained Convolutional Neural Network (CNN) fine-tuned for DIFR. During transfer learning, a pre-trained CNN learns generalized disguise-invariant features from facial images of several subjects to correctly identify them under varying facial disguises. We have compared four different pre-trained 2D CNNs, each with different number of learning parameters, based on their classification accuracy and execution time for selecting a suitable model for DIFR. Comprehensive experiments and comparative analysis have been conducted on six challenging facial disguise datasets. Resnet-18 gives the best trade-off between accuracy and efficiency, by achieving an average accuracy of 98.19% with an average execution time of 0.32 seconds. The promising results achieved in these experiments reflect the efficiency of the proposed method and outperforms the existing methods in all aspects.

Automatic object annotation in streamed and remotely explored large 3D reconstructions

We introduce a novel framework for 3D scene reconstruction with simultaneous object annotation, using a pre-trained 2D convolutional neural network (CNN), incremental data streaming, and remote exploration, with a virtual reality setup. It enables versatile integration of any 2D box detection or segmentation network. We integrate new approaches to (i) asynchronously perform dense 3D-reconstruction and object annotation at interactive frame rates, (ii) efficiently optimize CNN results in terms of object prediction and spatial accuracy, and (iii) generate computationally-efficient colliders in large triangulated 3D-reconstructions at run-time for 3D scene interaction. Our method is novel in combining CNNs with long and varying inference time with live 3D-reconstruction from RGB-D camera input. We further propose a lightweight data structure to store the 3D-reconstruction data and object annotations to enable fast incremental data transmission for real-time exploration with a remote client, which has not been presented before. Our framework achieves update rates of 22 fps (SSD Mobile Net) and 19 fps (Mask RCNN) for indoor environments up to 800 m3. We evaluated the accuracy of 3D-object detection. Our work provides a versatile foundation for semantic scene understanding of large streamed 3D-reconstructions, while being independent from the CNN’s processing time. Source code is available for non-commercial use.

Efficient Anomaly Detection in Crowd Videos Using Pre-Trained 2D Convolutional Neural Networks

Surveillance of crowded places can benefit from improved techniques of anomaly detection in crowd videos. Several existing methods have detected various types of crowd abnormal behaviors by using spatial and temporal information got from videos. So far as real-time detection of anomalies is concerned, special attention must be given to reducing the model complexity that leads to computational and memory loads. This paper proposes a low computational cost approach to detect crowd anomalies. The proposed approach avoids the expensive optical flow calculations by adopting a pre-trained 2D convolutional neural network (CNN) for motion information and implements a lighter form of the 2D CNN to achieve high recognition accuracy at low computational cost. Experiments on public datasets show that the proposed model outperforms the existing approaches in terms of detection accuracy alongside providing better performance in generating input frames.

VI-Net-View-Invariant Quality of Human Movement Assessment.

We propose a view-invariant method towards the assessment of the quality of human movements which does not rely on skeleton data. Our end-to-end convolutional neural network consists of two stages, where at first a view-invariant trajectory descriptor for each body joint is generated from RGB images, and then the collection of trajectories for all joints are processed by an adapted, pre-trained 2D convolutional neural network (CNN) (e.g., VGG-19 or ResNeXt-50) to learn the relationship amongst the different body parts and deliver a score for the movement quality. We release the only publicly-available, multi-view, non-skeleton, non-mocap, rehabilitation movement dataset (QMAR), and provide results for both cross-subject and cross-view scenarios on this dataset. We show that VI-Net achieves average rank correlation of 0.66 on cross-subject and 0.65 on unseen views when trained on only two views. We also evaluate the proposed method on the single-view rehabilitation dataset KIMORE and obtain 0.66 rank correlation against a baseline of 0.62.

Semi-CNN Architecture for Effective Spatio-Temporal Learning in Action Recognition

This paper introduces a fusion convolutional architecture for efficient learning of spatio-temporal features in video action recognition. Unlike 2D convolutional neural networks (CNNs), 3D CNNs can be applied directly on consecutive frames to extract spatio-temporal features. The aim of this work is to fuse the convolution layers from 2D and 3D CNNs to allow temporal encoding with fewer parameters than 3D CNNs. We adopt transfer learning from pre-trained 2D CNNs for spatial extraction, followed by temporal encoding, before connecting to 3D convolution layers at the top of the architecture. We construct our fusion architecture, semi-CNN, based on three popular models: VGG-16, ResNets and DenseNets, and compare the performance with their corresponding 3D models. Our empirical results evaluated on the action recognition dataset UCF-101 demonstrate that our fusion of 1D, 2D and 3D convolutions outperforms its 3D model of the same depth, with fewer parameters and reduces overfitting. Our semi-CNN architecture achieved an average of 16–30% boost in the top-1 accuracy when evaluated on an input video of 16 frames.

Action Recognition in Videos Using Pre-Trained 2D Convolutional Neural Networks

A pre-trained 2D CNN (Convolutional Neural Network) can be used for the spatial stream in the two-stream CNN structure for videos, treating the representative frame selected from the video as an input. However, the CNN for the temporal stream in the two-stream CNN needs training from scratch using the optical flow frames, which demands expensive computations. In this paper, we propose to adopt a pre-trained 2D CNN for the temporal stream to avoid the optical flow computations. Specifically, three RGB frames selected at three different times in the video sequence are converted into grayscale images and are assigned to three R(red), G(green), and B(blue) channels, respectively, to form a Stacked Grayscale 3-channel Image (SG3I). Then, the pre-trained 2D CNN is fine-tuned by SG3Is for the temporal stream CNN. Therefore, only pre-trained 2D CNNs are used for both spatial and temporal streams. To learn long-range temporal motions in videos, we can use multiple SG3Is by partitioning the video shot into sub-shots and a single SG3I is generated for each sub-shot. Experimental results show that our two-stream CNN with the proposed SG3Is is about 14.6 times faster than the first version of the two-stream CNN with the optical flow, and yet achieves a similar recognition accuracy for UCF-101 and a 5.7% better result for HMDB-51.

A robust and efficient convolutional deep learning framework for age‐invariant face recognition

AbstractExtensive research has been carried out in the past on face recognition, face detection, and age estimation. However, age‐invariant face recognition (AIFR) has not been explored that thoroughly. The facial appearance of a person changes considerably over time that results in introducing significant intraclass variations, which makes AIFR a very challenging task. Most of the face recognition studies that have addressed the ageing problem in the past have employed complex models and handcrafted features with strong parametric assumptions. In this work, we propose a novel deep learning framework that extracts age‐invariant and generalized features from facial images of the subjects. The proposed model trained on facial images from a minor part (20–30%) of lifespan of subjects correctly identifies them throughout their lifespan. A variety of pretrained 2D convolutional neural networks are compared in terms of accuracy, time, and computational complexity to select the most suitable network for AIFR. Extensive experimental results are carried out on the popular and challenging face and gesture recognition network ageing dataset. The proposed method achieves promising results and outperforms the state‐of‐the‐art AIFR models by achieving an accuracy of 99%, which proves the effectiveness of deep learning in facial ageing research.