Multiple Convolutional Neural Networks Research Articles

Multilevel Cloud Detection for High-Resolution Remote Sensing Imagery Using Multiple Convolutional Neural Networks

In high-resolution image data, multilevel cloud detection is a key task for remote sensing data processing. Generally, it is difficult to obtain high accuracy for multilevel cloud detection when using satellite imagery which only contains visible and near-infrared spectral bands. So, multilevel cloud detection for high-resolution remote sensing imagery is challenging. In this paper, a new multilevel cloud detection technique is proposed based on the multiple convolutional neural networks for high-resolution remote sensing imagery. In order to avoid input the entire image into the network for cloud detection, the adaptive simple linear iterative clustering (A-SCLI) algorithm was applied to the segmentation of the satellite image to obtain good-quality superpixels. After that, a new multiple convolutional neural networks (MCNNs) architecture is designed to extract multiscale features from each superpixel, and the superpixels are marked as thin cloud, thick cloud, cloud shadow, and non-cloud. The results suggest that the proposed method can detect multilevel clouds and obtain a high accuracy for high-resolution remote sensing imagery.

Fast GPU Implementation of a Scan-Specific Deep Learning Reconstruction for Accelerated Magnetic Resonance Imaging.

RAKI is a novel fast MRI image reconstruction algorithm that has been recently proposed, which gives satisfying results for highly accelerated MRI. However, due to RAKI reconstruction depends on multiple convolutional neural networks, implementing RAKI reconstruction is a time-consuming task. In this study, we present accelerate strategies for RAKI implementation aided by GPU parallel programming. Aiming at the characteristics of RAKI, we limited the iteration number of solving optimization problems in the network training stage, while maintaining the reconstruction results are visually satisfying. Further more, according to the independence between multiple networks, we parallelized the training tasks by CPU multiprocessing, which maximizes the performance by fully utilizing GPU resources. According to our experiments, these efforts gave more than 60x speed up compared with conventional, sequential implementation. With the ability of completing RAKI reconstruction in minutes, we are able to bring RAKI into practical applications.

Aerial Image Super Resolution via Wavelet Multiscale Convolutional Neural Networks

We develop an aerial image super-resolution method by training convolutional neural networks (CNNs) with respect to wavelet analysis. To this end, we commence by performing wavelet decomposition to aerial images for multiscale representations. We then train multiple CNNs for approximating the wavelet multiscale representations, separately. The multiple CNNs thus trained characterize aerial images in multiple directions and multiscale frequency bands, and thus enable image restoration subject to sophisticated culture variability. For inference, the trained CNNs regress wavelet multiscale representations from a low-resolution aerial image, followed by wavelet synthesis that forms a restored high-resolution aerial image. Experimental results validate the effectiveness of our method for restoring complicated aerial images.

Training CNNs for 3-D Sign Language Recognition With Color Texture Coded Joint Angular Displacement Maps

Convolutional neural networks (CNNs) can be remarkably effective for recognizing two-dimensional and three-dimensional (3-D) actions. To further explore the potential of CNNs, we applied them in the recognition of 3-D motion-captured sign language (SL). The sign's 3-D spatio-temporal information of each sign was interpreted using joint angular displacement maps (JADMs), which encode the sign as a color texture image; JADMs were calculated for all joint pairs. Multiple CNN layers then capitalized on the differences between these images and identify discriminative spatio-temporal features. We then compared the performance of our proposed model against those of the state-of-the-art baseline models by using our own 3-D SL dataset and two other benchmark action datasets, namely, HDM05 and CMU.

Dehazing for Multispectral Remote Sensing Images Based on a Convolutional Neural Network With the Residual Architecture

Multispectral remote sensing images are often contaminated by haze, which causes low image quality. In this paper, a novel dehazing method based on a deep convolutional neural network (CNN) with the residual structure is proposed for multispectral remote sensing images. First, multiple CNN individuals with the residual structure are connected in parallel and each individual is used to learn a regression from the hazy image to the clear image. Then, the outputs of CNN individuals are fused with weight maps to produce the final dehazing result. In the designed network, the CNN individuals, mining multiscale haze features through multiscale convolutions, are trained using different levels of haze samples to achieve different dehazing abilities. In addition, the weight maps change with the haze distribution, and the fusion of the CNN individuals is adaptive. The designed network is end-to-end, and putting a hazy image into it, the clear scene can be restored. To train the network, a wavelength-dependent haze simulation method is proposed to generate labeled data, which can synthesize hazy multispectral images highly close to real conditions. Experimental results show that the proposed method can accurately remove the haze in each band of multispectral images under different scenes.

Effective android malware detection with a hybrid model based on deep autoencoder and convolutional neural network

Android security incidents occurred frequently in recent years. To improve the accuracy and efficiency of large-scale Android malware detection, in this work, we propose a hybrid model based on deep autoencoder (DAE) and convolutional neural network (CNN). First, to improve the accuracy of malware detection, we reconstruct the high-dimensional features of Android applications (apps) and employ multiple CNN to detect Android malware. In the serial convolutional neural network architecture (CNN-S), we use Relu, a non-linear function, as the activation function to increase sparseness and “dropout” to prevent over-fitting. The convolutional layer and pooling layer are combined with the full-connection layer to enhance feature extraction capability. Under these conditions, CNN-S shows powerful ability in feature extraction and malware detection. Second, to reduce the training time, we use deep autoencoder as a pre-training method of CNN. With the combination, deep autoencoder and CNN model (DAE-CNN) can learn more flexible patterns in a short time. We conduct experiments on 10,000 benign apps and 13,000 malicious apps. CNN-S demonstrates a significant improvement compared with traditional machine learning methods in Android malware detection. In details, compared with SVM, the accuracy with the CNN-S model is improved by 5%, while the training time using DAE-CNN model is reduced by 83% compared with CNN-S model.

Adaptive Feature Abstraction for Translating Video to Text

Previous models for video captioning often use the output from a specific layer of a Convolutional Neural Network (CNN) as video features. However, the variable context-dependent semantics in the video may make it more appropriate to adaptively select features from the multiple CNN layers. We propose a new approach for generating adaptive spatiotemporal representations of videos for the captioning task. A novel attention mechanism is developed, that adaptively and sequentially focuses on different layers of CNN features (levels of feature "abstraction"), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on three benchmark datasets: YouTube2Text, M-VAD and MSR-VTT. Along with visualizing the results and how the model works, these experiments quantitatively demonstrate the effectiveness of the proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantics.

Primitive activity recognition from short sequences of sensory data

Activity recognition from mobile device sensors and wearables is attracting more attention from the research community due to the widespread adoption of these devices and the unique opportunity they provide for understanding user’s behavior leading to novel services and improvements in the delivery of existing ones. Approaches to tackle this problem either rely on predefined statistical features of sensor data streams or feature learning with the latter providing higher accuracies in most cases. Deep learning methods proved more effective than traditional approaches to feature learning in multiple studies. This paper presents a novel end-to-end trainable deep architecture that utilizes multiple convolutional neural networks (CNN), late fusion and extensive layer bypassing. The proposed method can easily accommodate multiple sensors and signal representations. The proposed approach is validated on eight publicly available datasets using a variety of evaluation conditions showing that it outperforms state-of-the-art methods in six of them.

Real Time Object Detection using CNN

Achieving new heights in object detection and image classification was made possible because of Convolution Neural Network(CNN). However, compared to image classification the object detection tasks are more difficult to analyze, more energy consuming and computation intensive. To overcome these challenges, a novel approach is developed for real time object detection applications to improve the accuracy and energy efficiency of the detection process. This is achieved by integrating the Convolutional Neural Networks (CNN) with the Scale Invariant Feature Transform (SIFT) algorithm. Here, we obtain high accuracy output with small sample data to train the model by integrating the CNN and SIFT features. The proposed detection model is a cluster of multiple deep convolutional neural networks and hybrid CNN-SIFT algorithm. The reason to use the SIFT featureis to amplify the model‟s capacity to detect small data or features as the SIFT requires small datasets to detect objects. Our simulation results show better performance in accuracy when compared with the conventional CNN method. As the resources like RAM, graphic card, ROM, etc. are limited we propose a pipelined implementation on an aggregate Central Processing Unit(CPU) and Graphical Processing Unit(GPU) platform.

Automated curation of brand-related social media images with deep learning

This paper presents a work consisting in using deep convolutional neural networks (CNNs) to facilitate the curation of brand-related social media images. The final goal is to facilitate searching and discovering user-generated content (UGC) with potential value for digital marketing tasks. The images are captured in real time and automatically annotated with multiple CNNs. Some of the CNNs perform generic object recognition tasks while others perform what we call visual brand identity recognition. When appropriate, we also apply object detection, usually to discover images containing logos. We report experiments with 5 real brands in which more than 1 million real images were analyzed. In order to speed-up the training of custom CNNs we applied a transfer learning strategy. We examine the impact of different configurations and derive conclusions aiming to pave the way towards systematic and optimized methodologies for automatic UGC curation.

A Real-Time Method to Estimate Speed of Object Based on Object Detection and Optical Flow Calculation

In recent years Convolutional Neural Network (CNN) has been widely used in computer vision field and makes great progress in lots of contents like object detection and classification. Even so, combining Convolutional Neural Network, which means making multiple CNN frameworks working synchronously and sharing their output information, could figure out useful message that each of them cannot provide singly. Here we introduce a method to real-time estimate speed of object by combining two CNN: YOLOv2 and FlowNet. In every frame, YOLOv2 provides object size; object location and object type while FlowNet providing the optical flow of whole image. On one hand, object size and object location help to select out the object part of optical flow image thus calculating out the average optical flow of every object. On the other hand, object type and object size help to figure out the relationship between optical flow and true speed by means of optics theory and priori knowledge. Therefore, with these two key information, speed of object can be estimated. This method manages to estimate multiple objects at real-time speed by only using a normal camera even in moving status, whose error is acceptable in most application fields like manless driving or robot vision.

Interleaved 3D-CNNs for joint segmentation of small-volume structures in head and neck CT images.

Accurate 3D image segmentation is a crucial step in radiation therapy planning of head and neck tumors. These segmentation results are currently obtained by manual outlining of tissues, which is a tedious and time-consuming procedure. Automatic segmentation provides an alternative solution, which, however, is often difficult for small tissues (i.e., chiasm and optic nerves in head and neck CT images) because of their small volumes and highly diverse appearance/shape information. In this work, we propose to interleave multiple 3D Convolutional Neural Networks (3D-CNNs) to attain automatic segmentation of small tissues in head and neck CT images. A 3D-CNN was designed to segment each structure of interest. To make full use of the image appearance information, multiscale patches are extracted to describe the center voxel under consideration and then input to the CNN architecture. Next, as neighboring tissues are often highly related in the physiological and anatomical perspectives, we interleave the CNNs designated for the individual tissues. In this way, the tentative segmentation result of a specific tissue can contribute to refine the segmentations of other neighboring tissues. Finally, as more CNNs are interleaved and cascaded, a complex network of CNNs can be derived, such that all tissues can be jointly segmented and iteratively refined. Our method was validated on a set of 48 CT images, obtained from the Medical Image Computing and Computer Assisted Intervention (MICCAI) Challenge 2015. The Dice coefficient (DC) and the 95% Hausdorff Distance (95HD) are computed to measure the accuracy of the segmentation results. The proposed method achieves higher segmentation accuracy (with the average DC: 0.58±0.17 for optic chiasm, and 0.71±0.08 for optic nerve; 95HD: 2.81±1.56mm for optic chiasm, and 2.23±0.90mm for optic nerve) than the MICCAI challenge winner (with the average DC: 0.38 for optic chiasm, and 0.68 for optic nerve; 95HD: 3.48 for optic chiasm, and 2.48 for optic nerve). An accurate and automatic segmentation method has been proposed for small tissues in head and neck CT images, which is important for the planning of radiotherapy.

Fault Diagnosis for Rotating Machinery Using Multiple Sensors and Convolutional Neural Networks

This paper presents a convolutional neural network (CNN) based approach for fault diagnosis of rotating machinery. The proposed approach incorporates sensor fusion by taking advantage of the CNN structure to achieve higher and more robust diagnosis accuracy. Both temporal and spatial information of the raw data from multiple sensors is considered during the training process of the CNN. Representative features can be extracted automatically from the raw signals. It avoids manual feature extraction or selection, which relies heavily on prior knowledge of specific machinery and fault types. The effectiveness of the developed method is evaluated by using datasets from two types of typical rotating machinery, roller bearings, and gearboxes. Compared with traditional approaches using manual feature extraction, the results show the superior diagnosis performance of the proposed method. The present approach can be extended to fault diagnosis of other machinery with various types of sensors due to its end to end feature learning capability.

Multispectral embedding-based deep neural network for three-dimensional human pose recovery

Monocular image-based three-dimensional (3-D) human pose recovery aims to retrieve 3-D poses using the corresponding two-dimensional image features. Therefore, the pose recovery performance highly depends on the image representations. We propose a multispectral embedding-based deep neural network (MSEDNN) to automatically obtain the most discriminative features from multiple deep convolutional neural networks and then embed their penultimate fully connected layers into a low-dimensional manifold. This compact manifold can explore not only the optimum output from multiple deep networks but also the complementary properties of them. Furthermore, the distribution of each hierarchy discriminative manifold is sufficiently smooth so that the training process of our MSEDNN can be effectively implemented only using few labeled data. Our proposed network contains a body joint detector and a human pose regressor that are jointly trained. Extensive experiments conducted on four databases show that our proposed MSEDNN can achieve the best recovery performance compared with the state-of-the-art methods.

Ship Classification in TerraSAR-X Images With Convolutional Neural Networks

Synthetic aperture radar (SAR) is an important instrument for oceanographic observations, providing detailed information of oceans’ surface and artificial floating structures. Due to advances in SAR technology and deployment of new SAR satellites, an increasing amount of data is available, and the development of efficient classification systems based on deep learning is possible. A deep neural network has improved the state of the art in classification tasks of optical images, but its use in SAR classification problems has been less exploited. In this paper, a full workflow for SAR maritime targets detection and classification on TerraSAR-X high-resolution image is presented, and convolutional neural networks (CNNs) recently proposed in the literature are cross evaluated on a common data set composed of five maritime classes, namely, cargo, tanker, windmill, platform, and harbor structure. Based on experiments and tests, a multiple input resolution CNN model is proposed and its performance is evaluated. Our results indicate that CNNs are efficient models to perform maritime target classification in SAR images, and the combination of different input resolutions in the CNN model improves its ability to derive features, increasing the overall classification score.

Disease named entity recognition from biomedical literature using a novel convolutional neural network

BackgroundAutomatic disease named entity recognition (DNER) is of utmost importance for development of more sophisticated BioNLP tools. However, most conventional CRF based DNER systems rely on well-designed features whose selection is labor intensive and time-consuming. Though most deep learning methods can solve NER problems with little feature engineering, they employ additional CRF layer to capture the correlation information between labels in neighborhoods which makes them much complicated.MethodsIn this paper, we propose a novel multiple label convolutional neural network (MCNN) based disease NER approach. In this approach, instead of the CRF layer, a multiple label strategy (MLS) first introduced by us, is employed. First, the character-level embedding, word-level embedding and lexicon feature embedding are concatenated. Then several convolutional layers are stacked over the concatenated embedding. Finally, MLS strategy is applied to the output layer to capture the correlation information between neighboring labels.ResultsAs shown by the experimental results, MCNN can achieve the state-of-the-art performance on both NCBI and CDR corpora.ConclusionsThe proposed MCNN based disease NER method achieves the state-of-the-art performance with little feature engineering. And the experimental results show the MLS strategy’s effectiveness of capturing the correlation information between labels in the neighborhood.

Clustering-Oriented Multiple Convolutional Neural Networks for Single Image Super-Resolution

In contrast to the human visual system (HVS) that applies different processing schemes to visual information of different textural categories, most existing deep learning models for image super-resolution tend to exploit an indiscriminate scheme for processing one whole image. Inspired by the human cognitive mechanism, we propose a multiple convolutional neural network framework trained based on different textural clusters of image local patches. To this end, we commence by grouping patches into K clusters via K-means, which enables each cluster center to encode image priors of a certain texture category. We then train K convolutional neural networks for super-resolution based on the K clusters of patches separately, such that the multiple convolutional neural networks comprehensively capture the patch textural variability. Furthermore, each convolutional neural network characterizes one specific texture category and is used for restoring patches belonging to the cluster. In this way, the texture variation within a whole image is characterized by assigning local patches to their closest cluster centers, and the super-resolution of each local patch is conducted via the convolutional neural network trained by its cluster. Our proposed framework not only exploits the deep learning capability of convolutional neural networks but also adapts them to depict texture diversities for super-resolution. Experimental super-resolution evaluations on benchmark image datasets validate that our framework achieves state-of-the-art performance in terms of peak signal-to-noise ratio and structural similarity. Our multiple convolutional neural network framework provides an enhanced image super-resolution strategy over existing single-mode deep learning models.

Feature-Fused SAR Target Discrimination Using Multiple Convolutional Neural Networks

Target discrimination has been one of the hottest issues in the interpretation of synthetic aperture radar (SAR) images. However, the presence of speckle noise and the absence of robust features make SAR discrimination difficult to deal with. Recently, convolutional neural network has obtained state-of-the-art results in pattern recognition. In this letter, we propose a target discrimination framework that jointly uses intensity and edge information of SAR images. This framework contains three parts, namely, feature extraction block, feature fusion block, and final classification block. In addition, a novel feature fusion method that can preserve the spatial relationship of different features is introduced. Experimental results on the miniSAR data demonstrate the effectiveness of our method.

Convolutional Neural Network Training incorporating Rotation-Based Generated Patterns and Handwritten Numeral Recognition of Major Indian Scripts

ABSTRACTHandwritten numeral recognition has gained much interest in recent times because of its diverse application potentials. Bangla and Hindi are the two major languages in Indian subcontinent and a large number of population in vast land scape uses Bangla and Devnagari numeral scripts of these two languages. Well-performed handwritten numeral recognition system for Bangla and Devnagari is challenging because of similar shaped numerals in both scripts; few numerals differ from their similar ones with a very few variation even in printed form. In this study, convolutional neural network (CNN) based two different methods have been investigated for better recognition of Bangla and Devnagari handwritten numerals. Both the methods use rotation-based generated patterns along with ordinary patterns to train CNN but in two different modes. In multiple CNN case, three different training sets (one with ordinary patterns and two with clockwise and anti-clockwise rotation-based generated patterns) are prepared; three different CNNs are trained individually with each of these training sets; and their decisions are combined for final system decision. On the other hand, in the case of single CNN, combination of above three training sets is used to train one CNN. A moderated pre-processing is also employed while generating patterns from the scanned images. The proposed methods have been tested on prominent benchmark handwritten numeral datasets and have achieved remarkable recognition accuracies. The achieved recognition accuracies are found better than reported recognition accuracies of prominent existing methods; and such outperformance mounted proposed methods as better recognition systems. Moreover, CNN's performance improvement due to use of generated patterns has also been clearly identified from the presented experimental results.

Superpixel-Based Multiple Local CNN for Panchromatic and Multispectral Image Classification

Recently, very high resolution (VHR) panchromatic and multispectral (MS) remote-sensing images can be acquired easily. However, it is still a challenging task to fuse and classify these VHR images. Generally, there are two ways for the fusion and classification of panchromatic and MS images. One way is to use a panchromatic image to sharpen an MS image, and then classify a pan-sharpened MS image. Another way is to extract features from panchromatic and MS images, respectively, and then combine these features for classification. In this paper, we propose a superpixel-based multiple local convolution neural network (SML-CNN) model for panchromatic and MS images classification. In order to reduce the amount of input data for the CNN, we extend simple linear iterative clustering algorithm for segmenting MS images and generating superpixels. Superpixels are taken as the basic analysis unit instead of pixels. To make full advantage of the spatial-spectral and environment information of superpixels, a superpixel-based multiple local regions joint representation method is proposed. Then, an SML-CNN model is established to extract an efficient joint feature representation. A softmax layer is used to classify these features learned by multiple local CNN into different categories. Finally, in order to eliminate the adverse effects on the classification results within and between superpixels, we propose a multi-information modification strategy that combines the detailed information and semantic information to improve the classification performance. Experiments on the classification of Vancouver and Xi’an panchromatic and MS image data sets have demonstrated the effectiveness of the proposed approach.