Multiple Convolutional Layers Research Articles

Hierarchical spatial-aware Siamese network for thermal infrared object tracking

Most thermal infrared (TIR) tracking methods are discriminative, treating the tracking problem as a classification task. However, the objective of the classifier (label prediction) is not coupled to the objective of the tracker (location estimation). The classification task focuses on the between-class difference of the arbitrary objects, while the tracking task mainly deals with the within-class difference of the same objects. In this paper, we cast the TIR tracking problem as a similarity verification task, which is coupled well to the objective of the tracking task. We propose a TIR tracker via a Hierarchical Spatial-aware Siamese Convolutional Neural Network (CNN), named HSSNet. To obtain both spatial and semantic features of the TIR object, we design a Siamese CNN that coalesces the multiple hierarchical convolutional layers. Then, we propose a spatial-aware network to enhance the discriminative ability of the coalesced hierarchical feature. Subsequently, we train this network end to end on a large visible video detection dataset to learn the similarity between paired objects before we transfer the network into the TIR domain. Next, this pre-trained Siamese network is used to evaluate the similarity between the target template and target candidates. Finally, we locate the candidate that is most similar to the tracked target. Extensive experimental results on the benchmarks VOT-TIR 2015 and VOT-TIR 2016 show that our proposed method achieves favorable performance compared to the state-of-the-art methods.

Thermal Face Authentication with Convolutional Neural Network

Matching thermal face images as a method of biometric authentication has gained increasing interest because of its advantage of tracking a target object at night and in total darkness. Therefore, for security purposes, it has become highly favourable and has extensive applications, for instance, in video surveillance at night. The aim of this study is to present a simple and efficient deep learning model, which accurately predicts person identification. A pre-trained Convolutional Neural Network (CNN) is employed to extract the features of the multiple convolution layers of the low resolutions’ thermal infrared images. To run the program and evaluate the performance, we use a sample of 1500 resized thermal images, each with resolution 181×161 pixels. The sample comprises of images that were captured within different time-lapse and with diverse emotions, poses and lighting conditions. The proposed approach is effective compared to the state-of-the-art thermal face recognition algorithms and achieves impressive accuracy of 99.6% with less processing and training times.

Dilated residual encode–decode networks for image denoising

Owing to recent advancements, very deep convolutional neural networks (CNNs) have found application in image denoising. However, while deeper models lead to better restoration performance, they are marred by a high number of parameters and increased training difficulty. To address these issues, we propose a CNN-based framework, named dilated residual encode–decode networks (DRED-Net), for image denoising, which learns direct end-to-end mappings from corrupted images to obtain clean images using few parameters. Our proposed network consists of multiple layers of convolution and deconvolution operators; in addition, we use dilated convolutions to boost the performance of our network without increasing the depth of the model or its complexity. Extensive experiments on synthetic noisy images are conducted to evaluate DRED-Net, and the results are compared with those obtained using state-of-the-art denoising methods. Our experimental results show that DRED-Net leads to results comparable with those obtained using other state-of-the-art methods for image denoising tasks.

Deep Distillation Recursive Network for Remote Sensing Imagery Super-Resolution

Deep convolutional neural networks (CNNs) have been widely used and achieved state-of-the-art performance in many image or video processing and analysis tasks. In particular, for image super-resolution (SR) processing, previous CNN-based methods have led to significant improvements, when compared with shallow learning-based methods. However, previous CNN-based algorithms with simple direct or skip connections are of poor performance when applied to remote sensing satellite images SR. In this study, a simple but effective CNN framework, namely deep distillation recursive network (DDRN), is presented for video satellite image SR. DDRN includes a group of ultra-dense residual blocks (UDB), a multi-scale purification unit (MSPU), and a reconstruction module. In particular, through the addition of rich interactive links in and between multiple-path units in each UDB, features extracted from multiple parallel convolution layers can be shared effectively. Compared with classical dense-connection-based models, DDRN possesses the following main properties. (1) DDRN contains more linking nodes with the same convolution layers. (2) A distillation and compensation mechanism, which performs feature distillation and compensation in different stages of the network, is also constructed. In particular, the high-frequency components lost during information propagation can be compensated in MSPU. (3) The final SR image can benefit from the feature maps extracted from UDB and the compensated components obtained from MSPU. Experiments on Kaggle Open Source Dataset and Jilin-1 video satellite images illustrate that DDRN outperforms the conventional CNN-based baselines and some state-of-the-art feature extraction approaches.

Concise feature pyramid region proposal network for multi-scale object detection

Object detection is a hot research issue in the field of computer vision. Many methods focus on detecting large objects. And features of small objects are easily weakened or even disappeared after multiple convolution layers. So the detection rate of multi-scale objects is unsatisfied. Aiming at this problem, a concise feature pyramid region proposal network (CFPRPN) is proposed to address the problem of small objects detection in this paper without missing the large objects. In the process of object detection, we propose a new method of adjustment for the object location. So the balanced detection of multi-scale objects is realized. CFPRPN combines image pyramids and feature pyramids. An image pyramid consists of scaled versions of an image and the feature pyramids produce multiple layers’ feature maps. They are both conducive to capturing the feature information of small objects in deep convolutional networks. At the same time, proposals of overlapping sizes from different layers are applied to improve the recall rate of multi-scale objects. These series operations are beneficial for CFPRPN to extract better proposals. We experimentally prove that after adding the fine-tuning location, the detection rate of multi-scale object is further improved. The inspiring thing is that refining location method is suitable for most algorithms of object detection.

Robust Visual Tracking via Hierarchical Convolutional Features.

Visual tracking is challenging as target objects often undergo significant appearance changes caused by deformation, abrupt motion, background clutter and occlusion. In this paper, we propose to exploit the rich hierarchical features of deep convolutional neural networks to improve the accuracy and robustness of visual tracking. Deep neural networks trained on object recognition datasets consist of multiple convolutional layers. These layers encode target appearance with different levels of abstraction. For example, the outputs of the last convolutional layers encode the semantic information of targets and such representations are invariant to significant appearance variations. However, their spatial resolutions are too coarse to precisely localize the target. In contrast, features from earlier convolutional layers provide more precise localization but are less invariant to appearance changes. We interpret the hierarchical features of convolutional layers as a nonlinear counterpart of an image pyramid representation and explicitly exploit these multiple levels of abstraction to represent target objects. Specifically, we learn adaptive correlation filters on the outputs from each convolutional layer to encode the target appearance. We infer the maximum response of each layer to locate targets in a coarse-to-fine manner. To further handle the issues with scale estimation and re-detecting target objects from tracking failures caused by heavy occlusion or out-of-the-view movement, we conservatively learn another correlation filter, that maintains a long-term memory of target appearance, as a discriminative classifier. We apply the classifier to two types of object proposals: (1) proposals with a small step size and tightly around the estimated location for scale estimation; and (2) proposals with large step size and across the whole image for target re-detection. Extensive experimental results on large-scale benchmark datasets show that the proposed algorithm performs favorably against the state-of-the-art tracking methods.

An Ensemble Stacked Convolutional Neural Network Model for Environmental Event Sound Recognition

Convolutional neural networks (CNNs) with log-mel audio representation and CNN-based end-to-end learning have both been used for environmental event sound recognition (ESC). However, log-mel features can be complemented by features learned from the raw audio waveform with an effective fusion method. In this paper, we first propose a novel stacked CNN model with multiple convolutional layers of decreasing filter sizes to improve the performance of CNN models with either log-mel feature input or raw waveform input. These two models are then combined using the Dempster–Shafer (DS) evidence theory to build the ensemble DS-CNN model for ESC. Our experiments over three public datasets showed that our method could achieve much higher performance in environmental sound recognition than other CNN models with the same types of input features. This is achieved by exploiting the complementarity of the model based on log-mel feature input and the model based on learning features directly from raw waveforms.

Hierarchical objectness network for region proposal generation and object detection

Recent region proposal generation methods show a low Intersection-of-Union with the ground-truth boxes. Because they simply regress the coordinates of the bounding boxes by exploiting the single-layer output of convolutional neural networks. This paper proposes a hierarchical objectness network for region proposal generation and object detection to address the inaccurate localization problem. Instead of regressing the coordinates, we subtly localize the objects by predicting the stripe objectness, i.e., a group of probabilities reflecting the existence of the object in each location of the candidate proposal. Additionally, we construct the hierarchical features by reversely connecting multiple convolutional layers to detect objects with large-scale variations. Our experimental results demonstrate that our method performs better than the state-of-the-art region proposal generation methods in terms of recall. Moreover, by integrating with advanced object detection frameworks, our method achieves superior object detection results.

Her2Net: A Deep Framework for Semantic Segmentation and Classification of Cell Membranes and Nuclei in Breast Cancer Evaluation.

We present an efficient deep learning framework for identifying, segmenting, and classifying cell membranes and nuclei from human epidermal growth factor receptor-2 (HER2)-stained breast cancer images with minimal user intervention. This is a long-standing issue for pathologists because the manual quantification of HER2 is error-prone, costly, and time-consuming. Hence, we propose a deep learning-based HER2 deep neural network (Her2Net) to solve this issue. The convolutional and deconvolutional parts of the proposed Her2Net framework consisted mainly of multiple convolution layers, max-pooling layers, spatial pyramid pooling layers, deconvolution layers, up-sampling layers, and trapezoidal long short-term memory (TLSTM). A fully connected layer and a softmax layer were also used for classification and error estimation. Finally, HER2 scores were calculated based on the classification results. The main contribution of our proposed Her2Net framework includes the implementation of TLSTM and a deep learning framework for cell membrane and nucleus detection, segmentation, and classification and HER2 scoring. Our proposed Her2Net achieved 96.64% precision, 96.79% recall, 96.71% F-score, 93.08% negative predictive value, 98.33% accuracy, and a 6.84% false-positive rate. Our results demonstrate the high accuracy and wide applicability of the proposed Her2Net in the context of HER2 scoring for breast cancer evaluation.

Predicting perceptual quality of images in realistic scenario using deep filter banks

Classical image perceptual quality assessment models usually resort to natural scene statistic methods, which are based on an assumption that certain reliable statistical regularities hold on undistorted images and will be corrupted by introduced distortions. However, these models usually fail to accurately predict degradation severity of images in realistic scenarios since complex, multiple, and interactive authentic distortions usually appear on them. We propose a quality prediction model based on convolutional neural network. Quality-aware features extracted from filter banks of multiple convolutional layers are aggregated into the image representation. Furthermore, an easy-to-implement and effective feature selection strategy is used to further refine the image representation and finally a linear support vector regression model is trained to map image representation into images’ subjective perceptual quality scores. The experimental results on benchmark databases present the effectiveness and generalizability of the proposed model.

Robust real-time visual object tracking via multi-scale fully convolutional Siamese networks

Robust visual object tracking against occlusions and deformations is still very challenging task. To tackle these issues, existing Convolutional Neural Networks (CNNs) based trackers either fail to handle them or can just run in low speed. In this paper, we present a realtime tracker which is robust to occlusions and deformations based on a Region-based, Multi-Scale Fully Convolutional Siamese Network (R-MSFCN). In the proposed R-MSFCN, the information of regions is extracted separately by the proposition of position-sensitive score maps on multiple convolutional layers. Combining these score maps via adaptive weights leads to accurate location of the target on a new frame. The experiments illustrate that our method outperforms state-of-the-art approaches, and can handle the cases of object deformation and occlusion at about 31 FPS.

LensFlow: A Convolutional Neural Network in Search of Strong Gravitational Lenses

Abstract In this work, we present our machine learning classification algorithm for identifying strong gravitational lenses from wide-area surveys using convolutional neural networks; LensFlow. We train and test the algorithm using a wide variety of strong gravitational lens configurations from simulations of lensing events. Images are processed through multiple convolutional layers that extract feature maps necessary to assign a lens probability to each image. LensFlow provides a ranking scheme for all sources that could be used to identify potential gravitational lens candidates by significantly reducing the number of images that have to be visually inspected. We apply our algorithm to the HST/ACS i-band observations of the COSMOS field and present our sample of identified lensing candidates. The developed machine learning algorithm is more computationally efficient and complimentary to classical lens identification algorithms and is ideal for discovering such events across wide areas from current and future surveys such as LSST and WFIRST.

Machine Learning Approach for Spatially and Temporally Resolved PM2.5 Exposures in the Continental United States

Background/Aim: Deep learning is a class of machine learning algorithms. Convolutional neural network, a deep learning algorithm, has brought about breakthroughs in processing image, speech, audio and text. Inside it, a convolutional layer takes information from nearby pixels to create high-level abstraction to improve model performance. This abstraction capacity is what traditional air pollution modelling lacks: traditional air pollution modelling uses variable values at monitor stations (in-situ information) to establish relationship and make prediction. However, neighbouring information (e.g., nearby traffic volume, neighbouring land-use type) also impacts local PM2.5 measurements but is often ignored in previous modelling. Convolutional neural network can potentially take neighbouring information into account and improve model performance. Methods: We used a convolutional neural network with multiple predictors, including aerosol optical depth, chemical transport model outputs, land-use variables, meteorological variables, surface reflectance, absorbing aerosol index as predictors to model ground-level PM2.5 from monitoring stations. For each variable, we extract its values at monitor stations as well as neighbouring locations as input information of the neural network. We incorporated multiple convolutional layers, pooling layers and full connection layers inside the neural network to model complex relationship between variables. Results: Model performance on validation data set indicated a good performance with daily R2 = 0.84 and MSE = 2.94 µg/m3. Model performance also exhibited regional variations with higher model performance in the Eastern and Central U.S. than the Western U.S. The Model still performed well at low PM2.5 levels (<12 µg/m3). Prediction results indicated higher PM2.5 concentrations in the Eastern and Central U.S. Summer time had higher PM2.5 levels than other seasons. Conclusions: This study explored a deep learning approach to model air pollutions with high accuracy, which facilitates follow-up epidemiological studies. This study suggests wider application of deep learning techniques in the field of environmental epidemiology.

Fast SAR target recognition based on random convolution features and ensemble extreme learning machines

Deep convolution neural network has demonstrated excellent performance in target detection and recognition tasks. However, few training samples and optimization design of deep models are two main problems to be solved when applied to SAR target recognition. This paper proposes an algorithm for SAR target recognition by combination of two dimensional random convolution features and ensemble extreme learning machines. Firstly, two dimensional random convolution kernels with different widths are generated, and convolution and pooling operations are performed in input image to extract random convolution feature vectors. Secondly, random samplings based on ensemble learning theory are done for extracted feature vectors to improve generalization performance of classifier and reduce training time, and base classifiers are then trained by extreme learning machines (ELM). Finally, majority vote method is adopted to combine the classification results of base classifiers. SAR target recognition experiments based on MSTAR database were performed, and experimental results demonstrate that, training time has dropped by ten times due to fast training capability of ELM, and the proposed algorithm achieves comparable classification performance with deep-learning-based methods which use data augmentation and multiple convolution layers. The proposed algorithm has the advantages of easy implementation and fewer adjustable parameters, and improves classifier's generalization performance through adoption of ensemble learning ideas.

Segmentation of Corn Leaf Disease Based on Fully Convolution Neural Network

Aiming at the problem that traditional computer vision is susceptible to different illumination and complex background in the segmentation of corn leaf disease, a method of segmentation of corn leaf disease based on full convolution neural network is proposed. Firstly, pre-processing and data enhancement are performed on the acquired images to establish a training set and a test set for deep learning. Secondly, the training centralized image is input into the FCN, and the feature map is obtained through multiple convolution, pooling, and activation layers; The feature map is upsampled to obtain a feature map of the same size as the input image. Finally, the resolution of the segmented image is restored by the process of deconvolution, and the segmentation result is output. This method was used to segment the common corn leaf disease images. The results show that the method has better segmentation effect and the segmentation accuracy is over 96%.

Multi-Center Brain Imaging Classification Using a Novel 3D CNN Approach

With the development of brain imaging technology, increasing amounts of magnetic resonance imaging data are being acquired, and traditional computational analysis methods based on single sites and small samples are facing substantial challenges. Deep learning technology, which is born via artificial intelligence, has shown the powerful ability to solve the classification problem based on big data in many studies, while it has not been widely used in brain imaging classification. Herein, we utilized our proposed novel 3-D deep adding neural network to classify 6008 samples from the largest data sets in the brain imaging field collected from more than 61 centers. The proposed method utilizes multiple convolutional layers to extract gradient information in different orientations and combines spatial information at two scales via the adding operation. High accuracy (over 92.5%) was obtained with a standard fivefold cross-validation strategy, demonstrating that the proposed method can effectively handle big data classifications from multiple centers. Compared with some traditional classification methods and some deep learning architectures, the proposed method was more accurate, demonstrating its stronger power to classify data from multiple centers. Our cross-site classification results prove that the proposed method is robust when training on a data set and testing on another data set. To the best of our knowledge, this paper is the first to classify neuroimaging data on such a large scale from multiple centers with such high accuracy. With its improved performance in classification and transferable program codes, the proposed method can potentially be used in intelligent medical treatment strategies and clinical practices based on mobile terminal.

Convolutional Recurrent Deep Learning Model for Sentence Classification

As the amount of unstructured text data that humanity produces overall and on the Internet grows, so does the need to intelligently to process it and extract different types of knowledge from it. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been applied to natural language processing systems with comparative, remarkable results. The CNN is a noble approach to extract higher level features that are invariant to local translation. However, it requires stacking multiple convolutional layers in order to capture long-term dependencies, due to the locality of the convolutional and pooling layers. In this paper, we describe a joint CNN and RNN framework to overcome this problem. Briefly, we use an unsupervised neural language model to train initial word embeddings that are further tuned by our deep learning network, then, the pre-trained parameters of the network are used to initialize the model. At a final stage, the proposed framework combines former information with a set of feature maps learned by a convolutional layer with long-term dependencies learned via long-short-term memory. Empirically, we show that our approach, with slight hyperparameter tuning and static vectors, achieves outstanding results on multiple sentiment analysis benchmarks. Our approach outperforms several existing approaches in term of accuracy; our results are also competitive with the state-of-the-art results on the Stanford Large Movie Review data set with 93.3% accuracy, and the Stanford Sentiment Treebank data set with 48.8% fine-grained and 89.2% binary accuracy, respectively. Our approach has a significant role in reducing the number of parameters and constructing the convolutional layer followed by the recurrent layer as a substitute for the pooling layer. Our results show that we were able to reduce the loss of detailed, local information and capture long-term dependencies with an efficient framework that has fewer parameters and a high level of performance.

Deep Visual Attention Prediction.

In this paper, we aim to predict human eye fixation with view-free scenes based on an end-to-end deep learning architecture. Although convolutional neural networks (CNNs) have made substantial improvement on human attention prediction, it is still needed to improve the CNN-based attention models by efficiently leveraging multi-scale features. Our visual attention network is proposed to capture hierarchical saliency information from deep, coarse layers with global saliency information to shallow, fine layers with local saliency response. Our model is based on a skip-layer network structure, which predicts human attention from multiple convolutional layers with various reception fields. Final saliency prediction is achieved via the cooperation of those global and local predictions. Our model is learned in a deep supervision manner, where supervision is directly fed into multi-level layers, instead of previous approaches of providing supervision only at the output layer and propagating this supervision back to earlier layers. Our model thus incorporates multi-level saliency predictions within a single network, which significantly decreases the redundancy of previous approaches of learning multiple network streams with different input scales. Extensive experimental analysis on various challenging benchmark data sets demonstrate our method yields the state-of-the-art performance with competitive inference time.

Single image super-resolution using a deep encoder–decoder symmetrical network with iterative back projection

Image super-resolution (SR) usually refers to reconstructing a high resolution (HR) image from a low resolution (LR) image without losing high frequency details or reducing the image quality. Recently, image SR based on convolutional neural network (SRCNN) was proposed and has received much attention due to its end-to-end mapping simplicity and superior performance. This method, however, only using three convolution layers to learn the mapping from LR to HR, usually converges slowly and leads to the size of output image reducing significantly. To address these issues, in this work, we propose a novel deep encoder-decoder symmetrical neural network (DEDSN) for single image SR. This deep network is fully composed of symmetrical multiple layers of convolution and deconvolution and there is no pooling (down-sampling and up-sampling) operations in the whole network so that image details degradation occurred in traditional convolutional frameworks is prevented. Additionally, in view of the success of the iterative back projection (IBP) algorithm in image SR, we further combine DEDSN with IBP network realization in this work. The new DEDSN-IBP model introduces the down sampling version of the ground truth image and calculates the simulation error as the prior guidance. Experimental results on benchmark data sets demonstrate that the proposed DEDSN model can achieve better performance than SRCNN and the improved DEDSN-IBP outperforms the reported state-of-the-art methods.

Online Scale Adaptive Visual Tracking Based on Multilayer Convolutional Features.

Convolutional neural networks can efficiently exploit sophisticated hierarchical features which have different properties for visual tracking problem. In this paper, by using multilayer convolutional features jointly and constructing a scale pyramid, we propose an online scale adaptive tracking method. We construct two separate correlation filters for translation and scale estimations. The translation filters improve the accuracy of target localization by a weighted fusion of multiple convolutional layers. Meanwhile, the separate scale filters achieve the optimal and fast scale estimation by a scale pyramid. This design decreases the mutual errors of translation and scale estimations, and reduces computational complexity efficiently. Moreover, in order to solve the problem of tracking drifts due to the severe occlusion or serious appearance changes of the target, we present a new adaptive and selective update mechanism to update the translation filters effectively. Extensive experimental results show that our proposed method achieves the excellent overall performance compared with the state-of-the-art methods.