Trained Convolutional Neural Network Model Research Articles

Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization

Objective. Convolutional neural networks (CNNs) have proven successful as function approximators and have therefore been used for classification problems including electroencephalography (EEG) signal decoding for brain–computer interfaces (BCI). Artificial neural networks, however, are considered black boxes, because they usually have thousands of parameters, making interpretation of their internal processes challenging. Here we systematically evaluate the use of CNNs for EEG signal decoding and investigate a method for visualizing the CNN model decision process. Approach. We developed a CNN model to decode the covert focus of attention from EEG event-related potentials during object selection. We compared the CNN and the commonly used linear discriminant analysis (LDA) classifier performance, applied to datasets with different dimensionality, and analyzed transfer learning capacity. Moreover, we validated the impact of single model components by systematically altering the model. Furthermore, we investigated the use of saliency maps as a tool for visualizing the spatial and temporal features driving the model output. Main results. The CNN model and the LDA classifier achieved comparable accuracy on the lower-dimensional dataset, but CNN exceeded LDA performance significantly on the higher-dimensional dataset (without hypothesis-driven preprocessing), achieving an average decoding accuracy of 90.7% (chance level = 8.3%). Parallel convolutions, tanh or ELU activation functions, and dropout regularization proved valuable for model performance, whereas the sequential convolutions, ReLU activation function, and batch normalization components reduced accuracy or yielded no significant difference. Saliency maps revealed meaningful features, displaying the typical spatial distribution and latency of the P300 component expected during this task. Significance. Following systematic evaluation, we provide recommendations for when and how to use CNN models in EEG decoding. Moreover, we propose a new approach for investigating the neural correlates of a cognitive task by training CNN models on raw high-dimensional EEG data and utilizing saliency maps for relevant feature extraction.

Quantification of Analyte Concentration in the Single Molecule Regime Using Convolutional Neural Networks.

Single molecule (SM) detection represents the ultimate limit of chemical detection. Over the years, many experimental techniques have emerged with this capacity. Yet, SM detection and imaging methods produce large spectral data sets that benefit from chemometric methods. In particular, surface enhanced Raman scattering spectroscopy (SERS), with extensive applications in biosensing, is demonstrated to be particularly promising because Raman active molecules can be identified without recognition elements and is capable of SM detection. Yet quantification at ultralow analyte concentrations requiring detection of SM events remains an ongoing challenge, with the few existing methods requiring carefully developed calibration curves that must be redeveloped for each analyte molecule. In this work, we demonstrate that a convolutional neural network (CNN) model when applied to bundles of SERS spectra yields a robust, facile method for concentration quantification down to 10 fM using SM detection events. We further demonstrate that transfer learning, the process of reusing the weights of a trained CNN model, greatly reduces the amount of data required to train CNN models on new analyte molecules. These results point the way for unambiguous analysis of large spectral data sets and the use of SERS in important ultra low concentration chemical detection applications such as metabolomic profiling, water quality evaluation, and fundamental research.

Imaging Through Turbid Media With Vague Concentrations Based on Cosine Similarity and Convolutional Neural Network

Underwater imaging has been extensively studied to bypass the limitation aroused by scattering and absorption of water solutions. It is highly meaningful to the development of optical imaging, especially in turbid media. The existing methods developed for reconstruction of original images from speckle patterns are applied in a stable medium, which obstruct wider applications in unpredictable media. Hence, it is crucial to take changeable environments into consideration to circumvent the limits of the extant methods. In this paper, we propose a new approach based on cosine similarity for speckle classification and convolutional neural network (CNN) for the reconstruction. The targets are placed in variant densities of turbid water mixed with certain of milk, and their corresponding intensity speckle patterns are recorded by a camera. It is verified that utilization of cosine similarity for the classification of patterns recorded in changeable media ensures high fidelity for label predictions. For a speckle pattern obtained in totally unidentified media, it can make a prediction of the density which has a high probability of accuracy. We can exploit the classified density to automatically select the most appropriate datasets to train a CNN model and then make predictions in real time with the trained CNN model. The combined model presented in this paper is tolerant to the uncertainty of turbidity. Moreover, it guarantees high-accuracy pattern classification and high-quality image reconstruction. It is feasible for potential applications in harsh water solutions with unknown perturbations of concentrations.

An enhanced fault-detection method based on adaptive spectral decomposition and super-resolution deep learning

Coherence is widely used for detecting faults in reservoir characterization. However, faults detected from coherence may be contaminated by some other discontinuities (e.g., noise and stratigraphic features) that are unrelated to faults. To further improve the accuracy and efficiency of coherence, preprocessing or postprocessing techniques are required. We developed an enhanced fault-detection method with adaptive scale highlighting and high resolution, by combining adaptive spectral decomposition and super-resolution (SR) deep learning into coherence calculation. As a preprocessing technique, adaptive spectral decomposition is first proposed and applied on seismic data to get a dominant-frequency-optimized amplitude spectrum, which has features of scale focus and multiple resolution. Eigenstructure-based coherence with dip correction is then calculated to delineate fault discontinuities. Following the remarkable success of SR deep learning in image reconstruction, a convolutional neural network (CNN) model is built and it then takes fault-detection images as the input to achieve enhanced results. The effectiveness of our proposed method is validated on a seismic survey acquired from Eastern China. Examples demonstrate that coherence from adaptive amplitude spectrum without dip correction is comparable to the dip-corrected one from seismic amplitude data at a certain degree, and they even highlight the specific scale of fault targets. Comparing fault detections from adaptive spectrum and some specific-frequency components, it can be concluded that adaptive spectral-based coherence highlights the primary scale of faults at various depths with only one single volume of data, thus improving the interpretation efficiency and reducing storage cost. Furthermore, with the trained CNN model, the resolution and signal-to-noise ratio of coherence images are effectively improved and the continuity of detected fault is promisingly enhanced.

End-to-End Automatic Image Annotation Based on Deep CNN and Multi-Label Data Augmentation

Automatic image annotation is a key step in image retrieval and image understanding. In this paper, we present an end-to-end automatic image annotation method based on a deep convolutional neural network (CNN) and multi-label data augmentation. Different from traditional annotation models that usually perform feature extraction and annotation as two independent tasks, we propose an end-to-end automatic image annotation model based on deep CNN (E2E-DCNN). E2E-DCNN transforms the image annotation problem into a multi-label learning problem. It uses a deep CNN structure to carry out the adaptive feature learning before constructing the end-to-end annotation structure using multiple cross-entropy loss functions for training. It is difficult to train a deep CNN model using small-scale datasets or scale up multi-label datasets using traditional data augmentation methods; hence, we propose a multi-label data augmentation method based on Wasserstein generative adversarial networks (ML-WGAN). The ML-WGAN generator can approximate the data distribution of a single multi-label image. The images generated by ML-WGAN can assist in the reduction of the over-fitting problem of training a deep CNN model and enhance the generalization ability of the trained CNN model. We optimize the network structure by using deformable convolution and spatial pyramid pooling. We experiment the proposed E2E-DCNN model with data augmentation by the proposed ML-WGAN on several public datasets. The experimental results demonstrate that the proposed model outperforms the state-of-the-art automatic image annotation models.

Feature and Model Level Fusion of Pretrained CNN for Remote Sensing Scene Classification

Convolutional neural networks (CNN) have attracted tremendous attention in the remote sensing community due to its excellent performance in different domains. Especially for remote sensing scene classification, the CNN-based methods have brought a great breakthrough. However, it is not feasible to fully design and train a new CNN model for remote sensing scene classification, as this usually requires a large number of training samples and high computational costs. To alleviate these limitations of fully training a new model, some work attempts to use the pretrained CNN models as feature extractors to build feature representation of scene images for classification and has achieved impressive results. In this scheme, how to construct feature representation of scene image via the pretrained CNN model becomes the key process. Existing studies paid a little attention to build more discriminative feature representation by exploring the potential benefits of multilayer features from a single CNN model and different feature representations from multiple CNN models. To this end, this paper presents a fusion strategy to build the feature representation of the scene images by integrating multilayer features of a single pretrained CNN model, and extends it to a framework of multiple CNN models. For these purposes, a multiscale improved Fisher kernel coding method is used to build feature representation of the scene images on convolutional layers, and a feature fusion approach based on two feature subspace learning methods [principal component analysis (PCA)/spectral regression kernel discriminant analysis and PCA/spectral regression kernel locality preserving projection] is proposed to construct final fused features for scene classification. For validation and comparison purposes, the proposed approaches are evaluated with two challenging high-resolution remote sensing datasets and shows the competitive performance compared with existing state-of-the-art baselines such as fully trained CNN models, fine tuning CNN models, and other related works.

Motion classification for radar moving target via STFT and convolution neural network

Radar moving target detection (MTD) technology is a key technology in the field of radar signal processing. The MTD method of radar has a better performance when applied to uniform motion detection although it has limited performance in other aspects, and it is also difficult to distinguish the type of the moving target. This article presents a new method for detecting and classifying moving targets based on convolutional neural network, which uses the convolutional neural network to learn the motion characteristics of the moving target echo so that realises the detection and classification of moving targets. At first, the authors model the moving target echo. Then the short-time Fourier transform (STFT) is used to perform time–frequency analysis. The obtained time–frequency graph is used as the input of the convolutional neural network which can automatically learns moving target features through training. Finally, the authors use the trained convolutional neural network model to detect and classify moving targets. The simulation verifies that the method has a great improvement in the detection accuracy rate compared with the traditional MTD.

Deep‐learning‐based power distribution network switch action identification leveraging dynamic features of distributed energy resources

This study proposes a data-driven approach for identifying switch actions in power distribution networks. Simulated micro-phasor measurement unit data is utilised to train a convolutional neural network (CNN) model. The trained CNN model can identify multi-phase multi-switch actions. Instead of working as a blackbox, the proposed approach extracts the features from the hidden layers of the trained CNN for engineering interpretation and error check. In addition, a random-forest-based feature ranking algorithm is proposed to identify the most important features. The proposed approach is validated on the IEEE 123-node feeder modelled in GridLAB-D. The CNN model is built and trained using TensorFlow. The proposed approach achieves 96.57% identification accuracy.

Wireless Indoor Localization Using Convolutional Neural Network and Gaussian Process Regression.

This paper presents a localization model employing convolutional neural network (CNN) and Gaussian process regression (GPR) based on Wi-Fi received signal strength indication (RSSI) fingerprinting data. In the proposed scheme, the CNN model is trained by a training dataset. The trained model adapts to complex scenes with multipath effects or many access points (APs). More specifically, the pre-processing algorithm makes the RSSI vector which is formed by considerable RSSI values from different APs readable by the CNN algorithm. The trained CNN model improves the positioning performance by taking a series of RSSI vectors into account and extracting local features. In this design, however, the performance is to be further improved by applying the GPR algorithm to adjust the coordinates of target points and offset the over-fitting problem of CNN. After implementing the hybrid model, the model is experimented with a public database that was collected from a library of Jaume I University in Spain. The results show that the hybrid model has outperformed the model using k-nearest neighbor (KNN) by 61.8%. While the CNN model improves the performance by 45.8%, the GPR algorithm further enhances the localization accuracy. In addition, the paper has also experimented with the three kernel functions, all of which have been demonstrated to have positive effects on GPR.

Deep-learning for seizure forecasting in canines with epilepsy

Objective. This paper introduces a fully automated, subject-specific deep-learning convolutional neural network (CNN) system for forecasting seizures using ambulatory intracranial EEG (iEEG). The system was tested on a hand-held device (Mayo Epilepsy Assist Device) in a pseudo-prospective mode using iEEG from four canines with naturally occurring epilepsy. Approach. The system was trained and tested on 75 seizures collected over 1608 d utilizing a genetic algorithm to optimize forecasting hyper-parameters (prediction horizon (PH), median filter window length, and probability threshold) for each subject-specific seizure forecasting model. The trained CNN models were deployed on a hand-held tablet computer and tested on testing iEEG datasets from four canines. The results from the iEEG testing datasets were compared with Monte Carlo simulations using a Poisson random predictor with equal time in warning to evaluate seizure forecasting performance. Main results. The results show the CNN models forecasted seizures at rates significantly above chance in all four dogs (p < 0.01, with mean 0.79 sensitivity and 18% time in warning). The deep learning method presented here surpassed the performance of previously reported methods using computationally expensive features with standard machine learning methods like logistic regression and support vector machine classifiers. Significance. Our findings principally support the feasibility of deploying trained CNN models on a hand-held computational device (Mayo Epilepsy Assist Device) that analyzes streaming iEEG data for real-time seizure forecasting.

Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision

In aquaculture, information on fish appetite is of great significance for guiding feeding and production practices. However, most fish appetite assessment methods are inefficient and subjective. To solve these problems, in this study, an automatic method for grading fish feeding intensity based on a convolutional neural network (CNN) and machine vision is proposed to evaluate fish appetite. The specific implementation process was as follows. First, images were collected during the feeding process, and a dataset was constructed and extended using rotation, scale, and translation (RST) augmentation techniques and noise-invariant data expansion. Then, a CNN was trained on the training dataset, and the fish appetite levels were graded using the trained CNN model. Finally, the performance of the method was evaluated and compared with other quantitative and qualitative feeding intensity assessment methods. The results show that the grading accuracy reached 90%; thus, the model can be used to detect and evaluate fish appetite to guide production practices.

Salt Delineation From Electromagnetic Data Using Convolutional Neural Networks

With recent advances in machine learning, convolutional neural networks (CNNs) have been successfully applied in many fields, and several attempts have been made in the field of geophysics. In this letter, we investigated the mapping of subsurface electrical resistivity distributions from electromagnetic (EM) data with CNNs. To begin imaging electrical resistivity using CNNs, we carried out precise delineation of a subsurface salt structure, which is indispensable for identification of offshore hydrocarbon reservoirs, using towed streamer EM data. For training the CNN model, an electrical resistivity model, including a salt body, and corresponding EM data calculated through numerical modeling were used as the label and input, respectively. The optimal weights and biases of the CNN were obtained minimizing the mean-square error between the predicted resistivity distribution and the target label. The final CNN model was selected using a validation data set during training. After training, we applied the trained CNN to test data sets of noisy data and simulated-SEAM data, which were not provided to the network during training. The test results demonstrate that our trained CNN model is stable, reliable, and efficient, and indicate the possibility of successful application of our CNN model to field data. Our study has shown the promising potential of CNNs for identifying defined subsurface electrical resistivity structures that are difficult to find using conventional EM inversion.

Intelligent Labeling Based on Fisher Information for Medical Image Segmentation Using Deep Learning.

Deep convolutional neural networks (CNN) have recently achieved superior performance at the task of medical image segmentation compared to classic models. However, training a generalizable CNN requires a large amount of training data, which is difficult, expensive, and time-consuming to obtain in medical settings. Active Learning (AL) algorithms can facilitate training CNN models by proposing a small number of the most informative data samples to be annotated to achieve a rapid increase in performance. We proposed a new active learning method based on Fisher information (FI) for CNNs for the first time. Using efficient backpropagation methods for computing gradients together with a novel low-dimensional approximation of FI enabled us to compute FI for CNNs with a large number of parameters. We evaluated the proposed method for brain extraction with a patch-wise segmentation CNN model in two different learning scenarios: universal active learning and active semi-automatic segmentation. In both scenarios, an initial model was obtained using labeled training subjects of a source data set and the goal was to annotate a small subset of new samples to build a model that performs well on the target subject(s). The target data sets included images that differed from the source data by either age group (e.g. newborns with different image contrast) or underlying pathology that was not available in the source data. In comparison to several recently proposed AL methods and brain extraction baselines, the results showed that FI-based AL outperformed the competing methods in improving the performance of the model after labeling a very small portion of target data set (<0.25%).

Dense motion estimation of particle images via a convolutional neural network

In this paper, we propose a supervised learning strategy for the fluid motion estimation problem (i.e., extracting the velocity fields from particle images). The purpose of this work is to design a convolutional neural network (CNN) for estimating dense motion field for particle image velocimetry (PIV), which allows to improve the computational efficiency without reducing the accuracy. First, the network model is developed based on FlowNetS, which is recently proposed for end-to-end optical flow estimation in the computer vision community. The input of the network is a particle image pair and the output is a velocity field with displacement vectors at every pixel. Second, a synthetic dataset of fluid flow images is generated to train the CNN model. To our knowledge, this is the first time a CNN has been used as a global motion estimator for particle image velocimetry. Experimental evaluations indicate that the trained CNN model can provide satisfactory results in both artificial and laboratory PIV images. The proposed estimator is also applied to the experiment of turbulent boundary layer. In addition, the computational efficiency of the CNN estimator is much superior to those of the traditional cross-correction and optical flow methods.

11K Hands: Gender recognition and biometric identification using a large dataset of hand images

Human hand not only possesses distinctive feature for gender information, it is also considered one of the primary biometric traits used to identify a person. Unlike face images, which are usually unconstrained, an advantage of hand images is they are usually captured under a controlled position. Most state-of-the-art methods, that rely on hand images for gender recognition or biometric identification, employ handcrafted features to train an off-the-shelf classifier or be used by a similarity metric for biometric identification. In this work, we propose a deep learning-based method to tackle the gender recognition and biometric identification problems. Specifically, we design a two-stream convolutional neural network (CNN) which accepts hand images as input and predicts gender information from these hand images. This trained model is then used as a feature extractor to feed a set of support vector machine classifiers for biometric identification. As part of this effort, we propose a large dataset of human hand images, 11K Hands, which contains dorsal and palmar sides of human hand images with detailed ground-truth information for different problems including gender recognition and biometric identification. By leveraging thousands of hand images, we could effectively train our CNN-based model achieving promising results. One of our findings is that the dorsal side of human hands is found to have effective distinctive features similar to, if not better than, those available in the palmar side of human hand images. To facilitate access to our 11K Hands dataset, the dataset, the trained CNN models, and our Matlab source code are available at ( https://goo.gl/rQJndd ).

Classification of myocardial infarction with multi-lead ECG signals and deep CNN

Myocardial infarction (MI), commonly known as heart attack, causes irreversible damage to heart muscles and even leads to death. Rapid and accurate diagnosis of MI is critical to avoid death. Blood tests and electrocardiogram (ECG) signals are used to diagnose acute MI. However, for an increase in blood enzyme values, a certain time must pass after the attack. This time lag may delay MI diagnosis. Hence, ECG diagnosis is still very important. Manual ECG interpretation requires expertise and is prone to inter-observer variability. Therefore, computer aided diagnosis may be useful in automatic detection of MI on ECG. In this study, a deep learning model with an end-to-end structure on the standard 12-lead ECG signal for the diagnosis of MI is proposed. For this purpose, the most commonly used technique, convolutional neural network (CNN) is used. Our trained CNN model with the proposed architecture yielded impressive accuracy and sensitivity performance over 99.00% for MI diagnosis on all ECG lead signals. Thus, the proposed model has the potential to provide high performance on MI detection which can be used in wearable technologies and intensive care units.

Convolutional Neural Networks for Direct Inference of Pharmacokinetic Parameters: Application to Stroke Dynamic Contrast-Enhanced MRI.

Background and Purpose: The T1-weighted dynamic contrast enhanced (DCE)-MRI is an imaging technique that provides a quantitative measure of pharmacokinetic (PK) parameters characterizing microvasculature of tissues. For the present study, we propose a new machine learning (ML) based approach to directly estimate the PK parameters from the acquired DCE-MRI image-time series that is both more robust and faster than conventional model fitting.Materials and Methods: We specifically utilize deep convolutional neural networks (CNNs) to learn the mapping between the image-time series and corresponding PK parameters. DCE-MRI datasets acquired from 15 patients with clinically evident mild ischaemic stroke were used in the experiments. Training and testing were carried out based on leave-one-patient-out cross- validation. The parameter estimates obtained by the proposed CNN model were compared against the two tracer kinetic models: (1) Patlak model, (2) Extended Tofts model, where the estimation of model parameters is done via voxelwise linear and nonlinear least squares fitting respectively.Results: The trained CNN model is able to yield PK parameters which can better discriminate different brain tissues, including stroke regions. The results also demonstrate that the model generalizes well to new cases even if a subject specific arterial input function (AIF) is not available for the new data.Conclusion: A ML-based model can be used for direct inference of the PK parameters from DCE image series. This method may allow fast and robust parameter inference in population DCE studies. Parameter inference on a 3D volume-time series takes only a few seconds on a GPU machine, which is significantly faster compared to conventional non-linear least squares fitting.

Training a Camera to Perform Long-Distance Eye Tracking by Another Eye-Tracker

Appearance-based gaze estimation techniques have been greatly advanced in these years. However, using a single camera for appearance-based gaze estimation has been limited to short distance in previous studies. In addition, labeling of training samples has been a time-consuming and unfriendly step in previous appearance-based gaze estimation studies. To bridge these significant gaps, this paper presents a new long-distance gaze estimation paradigm: train a camera to perform eye tracking by another eye tracker, named Learning-based Single Camera eye tracker (LSC eye-tracker). In the training stage, the LSC eye-tracker simultaneously acquired gaze data by a commercial trainer eye tracker and face appearance images by a long-distance trainee camera, based on which deep convolutional neural network (CNN) models are utilized to learn the mapping from appearance images to gazes. In the application stage, the LSC eye-tracker works alone to predict gazes based on the acquired appearance images by the single camera and the trained CNN models. Our experimental results show that the LSC eye-tracker enables both population-based eye tracking and personalized eye tracking with promising accuracy and performance.

Spectrum Analysis of EEG Signals Using CNN to Model Patient’s Consciousness Level Based on Anesthesiologists’ Experience

One of the most challenging predictive data analysis efforts is an accurate prediction of depth of anesthesia (DOA) indicators which has attracted growing attention since it provides patients a safe surgical environment in case of secondary damage caused by intraoperative awareness or brain injury. However, many researchers put heavily handcraft feature extraction or carefully tailored feature engineering to each patient to achieve very high sensitivity and low false prediction rate for a particular dataset. This limits the benefit of the proposed approaches if a different dataset is used. Recently, representations learned using the deep convolutional neural network (CNN) for object recognition are becoming a widely used model of the processing hierarchy in the human visual system. The correspondence between models and brain signals that holds the acquired activity at high temporal resolution has been explored less exhaustively. In this paper, deep learning CNN with a range of different architectures is designed for identifying related activities from raw electroencephalography (EEG). Specifically, an improved short-time Fourier transform is used to stand for the time-frequency information after extracting the spectral images of the original EEG as input to CNN. Then CNN models are designed and trained to predict the DOA levels from EEG spectrum without handcrafted features, which presents an intuitive mapping process with high efficiency and reliability. As a result, the best trained CNN model achieved an accuracy of 93.50%, interpreted as CNN’s deep learning to approximate the DOA by senior anesthesiologists, which highlights the potential of deep CNN combined with advanced visualization techniques for EEG-based brain mapping.

Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning

In existing visual representation learning tasks, deep convolutional neural networks (CNNs) are often trained on images annotated with single tags, such as ImageNet. However, a single tag cannot describe all important contents of one image, and some useful visual information may be wasted during training. In this work, we propose to train CNNs from images annotated with multiple tags, to enhance the quality of visual representation of the trained CNN model. To this end, we build a large-scale multi-label image database with 18M images and 11K categories, dubbed Tencent ML-Images. We efficiently train the ResNet-101 model with multi-label outputs on Tencent ML-Images, taking 90 hours for 60 epochs, based on a large-scale distributed deep learning framework,i.e.,TFplus. The good quality of the visual representation of the Tencent ML-Images checkpoint is verified through three transfer learning tasks, including single-label image classification on ImageNet and Caltech-256, object detection on PASCAL VOC 2007, and semantic segmentation on PASCAL VOC 2012. The Tencent ML-Images database, the checkpoints of ResNet-101, and all the training codehave been released at https://github.com/Tencent/tencent-ml-images. It is expected to promote other vision tasks in the research and industry community.