Train Deep Convolutional Neural Networks Research Articles

Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing Its Gradient Estimator Bias.

Equilibrium Propagation is a biologically-inspired algorithm that trains convergent recurrent neural networks with a local learning rule. This approach constitutes a major lead to allow learning-capable neuromophic systems and comes with strong theoretical guarantees. Equilibrium propagation operates in two phases, during which the network is let to evolve freely and then “nudged” toward a target; the weights of the network are then updated based solely on the states of the neurons that they connect. The weight updates of Equilibrium Propagation have been shown mathematically to approach those provided by Backpropagation Through Time (BPTT), the mainstream approach to train recurrent neural networks, when nudging is performed with infinitely small strength. In practice, however, the standard implementation of Equilibrium Propagation does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of equilibrium propagation, inherent in the use of finite nudging, is responsible for this phenomenon and that canceling it allows training deep convolutional neural networks. We show that this bias can be greatly reduced by using symmetric nudging (a positive nudging and a negative one). We also generalize Equilibrium Propagation to the case of cross-entropy loss (by opposition to squared error). As a result of these advances, we are able to achieve a test error of 11.7% on CIFAR-10, which approaches the one achieved by BPTT and provides a major improvement with respect to the standard Equilibrium Propagation that gives 86% test error. We also apply these techniques to train an architecture with unidirectional forward and backward connections, yielding a 13.2% test error. These results highlight equilibrium propagation as a compelling biologically-plausible approach to compute error gradients in deep neuromorphic systems.

Deep Learning Denoising Applied to Regional Distance Seismic Data in Utah

ABSTRACTSeismic waveform data are generally contaminated by noise from various sources. Suppressing this noise effectively so that the remaining signal of interest can be successfully exploited remains a fundamental problem for the seismological community. To date, the most common noise suppression methods have been based on frequency filtering. These methods, however, are less effective when the signal of interest and noise share similar frequency bands. Inspired by source separation studies in the field of music information retrieval (Jansson et al., 2017) and a recent study in seismology (Zhu et al., 2019), we implemented a seismic denoising method that uses a trained deep convolutional neural network (CNN) model to decompose an input waveform into a signal of interest and noise. In our approach, the CNN provides a signal mask and a noise mask for an input signal. The short-time Fourier transform (STFT) of the estimated signal is obtained by multiplying the signal mask with the STFT of the input signal. To build and test the denoiser, we used carefully compiled signal and noise datasets of seismograms recorded by the University of Utah Seismograph Stations network. Results of test runs involving more than 9000 constructed waveforms suggest that on average the denoiser improves the signal-to-noise ratios (SNRs) by ∼5 dB, and that most of the recovered signal waveforms have high similarity with respect to the target waveforms (average correlation coefficient of ∼0.80) and suffer little distortion. Application to real data suggests that our denoiser achieves on average a factor of up to ∼2–5 improvement in SNR over band-pass filtering and can suppress many types of noise that band-pass filtering cannot. For individual waveforms, the improvement can be as high as ∼15 dB.

Deep Tone-Mapping Operator Using Image Quality Assessment Inspired Semi-Supervised Learning

Tone-mapping operator (TMO) is intended to convert high dynamic range (HDR) content into a lower dynamic range so that it can be displayed on a standard dynamic range (SDR) device. The tone-mapped result of HDR content is usually stored as SDR image. For different HDR scenes, traditional TMOs are able to obtain a satisfying SDR image only under manually fine-tuned parameters. In this paper, we address this problem by proposing a learning-based TMO using deep convolutional neural network (CNN). We explore different CNN structure and adopt multi-scale and multi-branch fully convolutional design. When training deep CNN, we introduce image quality assessments (IQA), specifically, tone-mapped image quality assessment, and implement it as semi-supervised loss terms. We discuss and prove the effectiveness of semi-supervised loss terms, CNN structure, data pre-processing, etc. by several experiments. Finally, we demonstrate that our approach can produce appealing results under diversified HDR scenes.

Entropy estimation of the fragments of chest X-ray images

The subject of this research is medical chest X-ray images. After fundamental pre-processing, the accumulated database of such images can be used for training deep convolutional neural networks that have become one of the most significant innovations in recent years. The trained network carries out preliminary binary classification of the incoming images and serve as an assistant to the radiotherapist. For this purpose, it is necessary to train the neural network to carefully minimize type I and type II errors. Possible approach towards improving the effectiveness of application of neural networks, by the criteria of reducing computational complexity and quality of image classification, is the auxiliary approaches: image pre-processing and preliminary calculation of entropy of the fragments. The article provides the algorithm for X-ray image pre-processing, its fragmentation, and calculation of the entropy of separate fragments. In the course of pre-processing, the region of lungs and spine is selected, which comprises approximately 30-40% of the entire image. Then the image is divided into the matrix of fragments, calculating the entropy of separate fragments in accordance with Shannon’s formula based pm the analysis of individual pixels. Determination of the rate of occurrence of each of the 255 colors allows calculating the total entropy. The use of entropy for detecting pathologies is based on the assumption that its values differ for separate fragments and overall picture of its distribution between the images with the norm and pathologies. The article analyzes the statistical values: standard deviation of error, dispersion. A fully connected neural network is used for determining the patterns in distribution of entropy and its statistical characteristics on various fragments of the chest X-ray image.

An Intelligent Diagnosis Method of the Working Conditions in Sucker-Rod Pump Wells Based on Convolutional Neural Networks and Transfer Learning

In recent years, deep learning models represented by convolutional neural networks have shown incomparable advantages in image recognition and have been widely used in various fields. In the diagnosis of sucker-rod pump working conditions, due to the lack of a large-scale dynamometer card data set, the advantages of a deep convolutional neural network are not well reflected, and its application is limited. Therefore, this paper proposes an intelligent diagnosis method of the working conditions in sucker-rod pump wells based on transfer learning, which is used to solve the problem of too few samples in a dynamometer card data set. Based on the dynamometer cards measured in oilfields, image classification and preprocessing are conducted, and a dynamometer card data set including 10 typical working conditions is created. On this basis, using a trained deep convolutional neural network learning model, model training and parameter optimization are conducted, and the learned deep dynamometer card features are transferred and applied so as to realize the intelligent diagnosis of dynamometer cards. The experimental results show that transfer learning is feasible, and the performance of the deep convolutional neural network is better than that of the shallow convolutional neural network and general fully connected neural network. The deep convolutional neural network can effectively and accurately diagnose the working conditions of sucker-rod pump wells and provide an effective method to solve the problem of few samples in dynamometer card data sets.

ЖАДНОЕ ПОСЛОЙНОЕ ОБУЧЕНИЕ СВЕРТОЧНЫХ НЕЙРОННЫХ СЕТЕЙ

Layer-by-layer training is an alternative end-to-end backpropagation approach for training deep convolutional neural networks. Layer-by-layer training on specific architectures can yield very competitive results. In the ImageNet database (www.imagenet. org), layer-by-layer trained networks can perform comparable to many modern end-to-end trained networks. This article compares the performance gap between the two training procedures over a wide range of network architectures and further analyzes the potential limitations of layer-by-layer training. The results show that layer-by-layer learning quickly saturates after a certain critical level due to overfitting of early levels in neural networks. Several approaches that have been used to solve this problem are discussed, and a methodology for improving layer-by-layer learning in various neural network architectures is discussed. Fundamentally, this research highlights the need to open up the black box that represents modern deep neural networks and explore the layer-by-layer interactions between intermediate hidden layers within deep networks through the lens of layer-by-layer learning.

A Novel CONV Acceleration Strategy Based on Logical PE Set Segmentation for Row Stationary Dataflow

Deep convolutional neural networks (DCNNs) have been proposed as enhanced developments of neural networks (NNs) in the field of artificial intelligence (AI) and successfully applied in deep learning (DL) scenarios. With the advancement of technology, the number of network layers has continuously increased, resulting in a huge number of calculations and memory accesses required in the training and inference process of DCNNs and thereby hindering their further deployment and application. Using a specific dataflow formed by reusable DCNN data in the network-on-chip (NoC), reducing the memory access pressure and improving DCNN processing efficiency has become a promising acceleration schemes for the current DCNN. In this paper, a novel convolution layer (CONV) acceleration strategy based on logical PE set segmentation for row stationary (RS) dataflow is proposed to solve the problems of low flexibility and inefficient processing array utilization faced by the conventional folding mapping strategy. The simulation results show that the new mapping strategy based on PE set segmentation can achieve better processing element utilization and CONV acceleration improvement at the expense of little increase in the data movement energy consumption compared with the conventional strategy.

Iris Recognition Based on Fine-Tune SquIrisNet

Convolutional neural networks (CNN) have achieved unprecedented results in the fields of pattern recognition and image processing. CNN methods have also been gradually applied to iris recognition. However, in the case of insufficient training samples, training deep CNN models is prone to overfitting. In order to solve the above problem, this paper proposes an iris recognition method based on fine-tune SquIrisNet. When the pre-trained SqueezNet is migrated to the iris dataset, the fully connected layers are used instead of the convolutional layer and the global pooling layer, then the parameters are adjusted using the error back propagation algorithm, and finally the images are classified by the Softmax classifier. The fully connected layers added in this model can play the role of “firewall” in the fine-tune process, retaining the model complexity to a certain extent. The experimental results on the IIT Delhi and SDUMLA-HMT iris databases show that the proposed method has higher correct recognition rate, faster convergence speed and better robustness than AlexNet and VGGNet.

Creating a Simulated Dataset for Training Deep Convolutional Neural Networks for Use in Cardiovascular Photoacoustic Tomography

Background/Objective: Photoacoustic tomography possesses increasing potential as a non-invasive imaging method that combines optical and acoustic imaging to maximize the visualization of tissue. Determining the composition, orientation, and location of anatomical structures in multidimensional space requires maximizing image resolution and differentiation from noise and reflection artifacts. Using simulations to develop and improve methods for image resolution allows for flexibility and variation of numerous variables. 
 
 Methods: Binary masks were created from mouse common carotid ultrasound images using a graphical user interface for MATLAB. With the k-Wave toolbox, we performed time-reversal photoacoustic simulations using the masks. Medium properties for the simulations were assigned for sound speed and density for connective tissue (1540 m/s, 1027 kg/m3) and arterial walls (1569 m/s, 1102 kg/m3). The dataset was augmented through rotational and mirrored transformations and the addition of noise and reflection artifacts via Python open-source software. 
 
 Results: A set of 87 binary masks was generated from common carotid ultrasound images. These masks were used to simulate initial pressure distributions through the k-Wave toolbox to reconstruct the structure of the common carotid. Each simulation yielded graphs for initial pressure and sensor distribution, simulated sensor data, reconstructed initial pressure, and a comparison profile between the original and reconstructed pressure. Data augmentation was implemented using the reconstructed pressure output from the 87 simulations, each producing 12 distinct images from rotations and mirroring with the addition of noise and reflection artifacts. The final dataset yielded 1044 images. 
 
 Conclusion and Potential Impact: Future work will involve applying this dataset to a neural network to improve photoacoustic quality such that transfer learning can be applied on ex vivo and in vivo datasets. Thus, there is potential for use in diagnostic applications in patients with cardiovascular disease states like atherosclerosis and aneurysms that require high resolution visualization of tissue structure and composition.

Learning Collision-Free Space Detection From Stereo Images: Homography Matrix Brings Better Data Augmentation

Collision-free space detection is a critical component of autonomous vehicle perception. The state-of-the-art algorithms are typically based on supervised deep learning. Their performance is dependent on the quality and amount of labeled training data. It remains an open challenge to train deep convolutional neural networks (DCNNs) using only a small quantity of training samples. Therefore, in this article, we mainly explore an effective training data augmentation approach that can be employed to improve the overall DCNN performance, when additional images captured from different views are available. Due to the fact that the pixels in collision-free space (generally regarded as a planar surface) between two images, captured from different views, can be associated using a homography matrix, the target image can be transformed into the reference view. This provides a simple but effective way to generate training data from additional multiview images. Extensive experimental results, conducted with six state-of-the-art semantic segmentation DCNNs on three datasets, validate the effectiveness of the proposed method for enhancing collision-free space detection performance. When validated on the KITTI road benchmark, our approach provides the best results, compared with other state-of-the-art stereo vision-based collision-free space detection approaches.

Towards a consistent framework of comparing galaxy mergers in observations and simulations

Aims.We aim to perform consistent comparisons between observations and simulations on the mass dependence of the galaxy major merger fraction at low redshift over an unprecedentedly wide range of stellar masses (∼109to 1012 M⊙).Methods.We first carry out forward modelling of ideal synthetic images of major mergers and non-mergers selected from the Next Generation Illustris Simulations (IllustrisTNG) to include major observational effects. We then train deep convolutional neural networks (CNNs) using realistic mock observations of galaxy samples from the simulations. Subsequently, we apply the trained CNNs to real the Kilo-Degree Survey (KiDS) images of galaxies selected from the Galaxy And Mass Assembly (GAMA) survey. Based on the major merger samples, which are detected in a consistent manner in the observations and simulations, we determine the dependence of major merger fraction on stellar mass atz ∼ 0.15 and make comparisons between the two.Results.The detected major merger fraction in the GAMA/KiDS observations has a fairly mild decreasing trend with increasing stellar mass over the mass range 109 M⊙ < M* < 1011.5 M⊙. There is good agreement in the mass dependence of the major merger fraction in the GAMA/KiDS observations and the IllustrisTNG simulations over 109.5 M⊙ < M* < 1010.5 M⊙. However, the observations and the simulations show some differences atM* > 1010.5 M⊙, possibly due to the supermassive blackhole feedback in its low-accretion state in the simulations which causes a sharp transition in the quenched fractions at this mass scale. The discrepancy could also be due to the relatively small volume of the simulations and/or differences in how stellar masses are measured in simulations and observations.

Damage identification using deep learning and long-gauge fiber Bragg grating sensors.

This study proposes an innovative structural damage identification method using dynamic response measured by long-gauge fiber Bragg grating (FBG) sensors and accelerometers to train deep convolutional neural networks (DCNNs). At the same time, the pre-trained model is applied to another structure through transfer learning (TL) technology. To verify this method, the I-shaped steel beam vibration test was conducted at first. Three types of data, acceleration, wavelength, and the fusion of the former, are utilized to train convolutional neural network (CNN) models, and the models are then tested and compared. In TL, another CNN is pre-trained using FBG data of steel beam. Next, the data of T-shaped reinforced concrete (RC) beam are employed to train the pre-trained model. The performance of the CNN is evaluated by training history and the confusion matrix. The results show the CNN-based damage identification method can classify the damage pattern accurately. The CNN trained by fusion data has both high classification accuracy and faster training speed. TL technology can greatly reduce the training time of other target tasks in the same fields.

A compensation-based optimization strategy for top dense layer training

The stochastic gradient descent (SGD) method plays a central role in training deep convolutional neural networks (DCNNs). The recent advances in the field of optimization methods for DCNNs follow the direction of gradients. The innovations mainly lie in adopting different techniques to manage the history of gradients or automatically adapt the step size. In contrast, in this paper we propose a novel optimization approach for training the top dense layer of DCNN. It primarily utilizes the orientation directly pointing to the optimal values of parameters instead of the direction of gradients to navigate the parameters updates. The Moore–Penrose inverse method has been adopted to determine the difference between the current parameters and the optimal parameters, and the parameter updates are driven along this direction to compensate such a difference. Subsequently, the parameters have been fine-tuned along the direction of the classical gradient. Experiments have been conducted on extensively selected benchmark datasets. The results indicate that the proposed approach obtains a higher convergence rate and lower minimum loss compared to other state-of-the-art optimization methods. Furthermore, with the same DCNN architectures, the performance improvement margin between the proposed optimization method and other state-of-the-art optimization approaches is highly significant.

Identification of sunflower seeds with deep convolutional neural networks

In the food and agricultural industries, it is crucial to identify and to choose correct sunflower seeds that meet specific requirements. Deep learning and computer vision methods can help identify sunflower seeds. In this study, a computer vision system was proposed, trained, and tested to identify four varieties of sunflower seeds using deep learning methodology and a regular color camera. Image acquisition was carried out under controlled illumination conditions. An image segmentation procedure was employed to reduce the workload in obtaining training images required for training deep convolutional neural network models. Three deep learning architectures, namely AlexNet, GoogleNet, and ResNet, were investigated for identifying sunflower seeds in this study. Different solver types were also evaluated to determine the best deep learning model in terms of both accuracy and training time. About 4800 sunflower seeds were inspected individually for training and testing. The highest classification accuracy (95%) was succeeded with the GoogleNet algorithm.

Teaching Neural Networks to Generate Fast Sunyaev–Zel’dovich Maps

Abstract The thermal Sunyaev–Zel’dovich (tSZ) and the kinematic Sunyaev–Zel’dovich (kSZ) effects trace the distribution of electron pressure and momentum in the hot universe. These observables depend on rich multiscale physics, thus, simulated maps should ideally be based on calculations that capture baryonic feedback effects such as cooling, star formation, and other complex processes. In this paper, we train deep convolutional neural networks with a U-Net architecture to map from the three-dimensional distribution of dark matter to electron density, momentum, and pressure at ∼100 kpc resolution. These networks are trained on a combination of the TNG300 volume and a set of cluster zoom-in simulations from the IllustrisTNG project. The neural nets are able to reproduce the power spectrum, one-point probability distribution function, bispectrum, and cross-correlation coefficients of the simulations more accurately than the state-of-the-art semianalytical models. Our approach offers a route to capture the richness of a full cosmological hydrodynamical simulation of galaxy formation with the speed of an analytical calculation.

Choosing an appropriate training set size when using existing data to train neural networks for land cover segmentation

ABSTRACT Land cover data is an inventory of objects on the Earth’s surface, which is often derived from remotely sensed imagery. Deep Convolutional Neural Network (DCNN) is a competitive method in image semantic segmentation. Some scholars argue that the inadequacy of training set is an obstacle when applying DCNNs in remote sensing image segmentation. While existing land cover data can be converted to large training sets, the size of training data set needs to be carefully considered. In this paper, we used different portions of a high-resolution land cover map to produce different sizes of training sets to train DCNNs (SegNet and U-Net) and then quantitatively evaluated the impact of training set size on the performance of the trained DCNN. We also introduced a new metric, Edge-ratio, to assess the performance of DCNN in maintaining the boundary of land cover objects. Based on the experiments, we document the relationship between the segmentation accuracy and the size of the training set, as well as the nonstationary accuracies among different land cover types. The findings of this paper can be used to effectively tailor the existing land cover data to training sets, and thus accelerate the assessment and employment of deep learning techniques for high-resolution land cover map extraction.

CADN: A weakly supervised learning-based category-aware object detection network for surface defect detection

Large-scale data with human annotations is of crucial importance for training deep convolutional neural network (DCNN) to ensure stable and reliable performance. However, accurate annotations, such as bounding box and pixel-level annotations, demand expensive labeling efforts, which has prevented wide application of DCNN in industries. Focusing on the problem of surface defect detection, this paper proposes a weakly supervised learning method named Category-Aware object Detection network (CADN) to tackle the dilemma. CADN is trained with image tag annotations only and performs image classification and defect localization simultaneously. The weakly supervised learning is achieved by extracting category-aware spatial information in a classification pipeline. CADN could be equipped with either a lighter or a larger backbone network as the feature extractor resulting in better real-time performance or higher accuracy. To address the two conflicting objectives simultaneously, both of which are significant concerns in industrial applications, knowledge distillation strategy is adopted to force the learned features of a lighter CADN to mimic that of a larger CADN. Accordingly, the accuracy of the lighter CADN is improved while high real-time performance is maintained. The proposed approach is verified on our own defect dataset as well as on an open-source defect dataset. As demonstrated, satisfied performance is achieved by the proposed method, which could meet industrial requirements completely. Meanwhile, the method minimizes human efforts involved in image labelling, thus promoting the applications of DCNN in industries.

Cloud Image Classification Method Based on Deep Convolutional Neural Network

Compared with satellite remote sensing images, ground-based invisible images have limited swath, but featured in higher resolution, more distinct cloud features, and the cost is greatly reduced, conductive to continuous meteorological observation of local areas. For the first time, this paper proposed a high-resolution cloud image classification method based on deep learning and transfer learning technology for ground-based invisible images. Due to the limited amount of samples, traditional classifiers such as support vector machine can't effectively extract the unique features of different types of clouds, and directly training deep convolutional neural networks leads to over-fitting. In order to prevent the network from over-fitting, this paper proposed applying transfer learning method to fine-tune the pre-training model. The proposed network achieved as high as 85.19% test accuracy on 6-type cloud images classification task. The networks proposed in this paper can be applied to classify digital photos captured by cameras directly, which will reduce the cost of system greatly.

Zero-Shot Learning Based on Deep Weighted Attribute Prediction

In zero-shot learning, attributes play as a bridge from original images to class labels. Therefore, to achieve accurate zero-shot image classification, we mainly focus on improving attribute prediction accuracy by taking full advantage of prior information about attribute from two aspects. First, we present a new attribute classifier called deep attribute prediction (DeepAP) model by using supervised deep convolutional neural networks (DCNNs), where the attribute label information participates in the training of DCNNs. Unlike common DCNNs that are usually used to extract image features, the constructed DCNNs are used to directly predict attribute values from the original input images. Thus, the designed DeepAP model can serve as the mapping from low-level image features to high-level semantic attributes in the traditional direct attribute prediction (DAP) model. Second, another prior information about attribute, i.e., class-attribute matrix is used to mine the attribute-class correlation with sparse representation coefficients. Since the attribute-class correlation can reflect different contributions of attributes to classification, we use it to define attribute weights and incorporate the idea of weighted attributes into DeepAP to form the deep weighted attribute prediction (DWAP) model. Experiments on three real datasets show that DWAP outperforms the deep attribute network and DAP on attribute prediction and zero-shot image classification.

Improved object recognition using neural networks trained to mimic the brain’s statistical properties

The current state-of-the-art object recognition algorithms, deep convolutional neural networks (DCNNs), are inspired by the architecture of the mammalian visual system, and are capable of human-level performance on many tasks. As they are trained for object recognition tasks, it has been shown that DCNNs develop hidden representations that resemble those observed in the mammalian visual system (Razavi and Kriegeskorte, 2014; Yamins and Dicarlo, 2016; Gu and van Gerven, 2015; Mcclure and Kriegeskorte, 2016). Moreover, DCNNs trained on object recognition tasks are currently among the best models we have of the mammalian visual system. This led us to hypothesize that teaching DCNNs to achieve even more brain-like representations could improve their performance. To test this, we trained DCNNs on a composite task, wherein networks were trained to: (a) classify images of objects; while (b) having intermediate representations that resemble those observed in neural recordings from monkey visual cortex. Compared with DCNNs trained purely for object categorization, DCNNs trained on the composite task had better object recognition performance and are more robust to label corruption. Interestingly, we found that neural data was not required for this process, but randomized data with the same statistical properties as neural data also boosted performance. While the performance gains we observed when training on the composite task vs the “pure” object recognition task were modest, they were remarkably robust. Notably, we observed these performance gains across all network variations we studied, including: smaller (CORNet-Z) vs larger (VGG-16) architectures; variations in optimizers (Adam vs gradient descent); variations in activation function (ReLU vs ELU); and variations in network initialization. Our results demonstrate the potential utility of a new approach to training object recognition networks, using strategies in which the brain – or at least the statistical properties of its activation patterns – serves as a teacher signal for training DCNNs.