Learning High-level Features Research Articles

Robust Visual Ship Tracking with an Ensemble Framework via Multi-View Learning and Wavelet Filter.

Maritime surveillance videos provide crucial on-spot kinematic traffic information (traffic volume, ship speeds, headings, etc.) for varied traffic participants (maritime regulation departments, ship crew, ship owners, etc.) which greatly benefits automated maritime situational awareness and maritime safety improvement. Conventional models heavily rely on visual ship features for the purpose of tracking ships from maritime image sequences which may contain arbitrary tracking oscillations. To address this issue, we propose an ensemble ship tracking framework with a multi-view learning algorithm and wavelet filter model. First, the proposed model samples ship candidates with a particle filter following the sequential importance sampling rule. Second, we propose a multi-view learning algorithm to obtain raw ship tracking results in two steps: extracting a group of distinct ship contour relevant features (i.e., Laplacian of Gaussian, local binary pattern, Gabor filter, histogram of oriented gradient, and canny descriptors) and learning high-level intrinsic ship features by jointly exploiting underlying relationships shared by each type of ship contour features. Third, with the help of the wavelet filter, we performed a data quality control procedure to identify abnormal oscillations in the ship positions which were further corrected to generate the final ship tracking results. We demonstrate the proposed ship tracker’s performance on typical maritime traffic scenarios through four maritime surveillance videos.

One-dimensional convolutional auto-encoder-based feature learning for fault diagnosis of multivariate processes

Noise and high-dimension of process signals decrease effectiveness of those regular fault detection and diagnosis models in multivariate processes. Deep learning technique shows very excellent performance in high-level feature learning from image and visual data. However, the large labeled data are required for deep neural networks (DNNs) with supervised learning like convolutional neural network (CNN), which increases the time cost of model construction significantly. A new DNN model, one-dimensional convolutional auto-encoder (1D-CAE) is proposed for fault detection and diagnosis of multivariate processes in this paper. 1D-CAE is utilized to learn hierarchical feature representations through noise reduction of high-dimensional process signals. Auto-encoder integrated with convolutional kernels and pooling units allows feature extraction to be particularly effective, which is of great importance for fault detection and diagnosis in multivariate processes. The comparison between 1D-CAE and other typical DNNs illustrates effectiveness of 1D-CAE for fault detection and diagnosis on Tennessee Eastman Process and Fed-batch fermentation penicillin process. The proposed method provides an effective platform for deep-learning-based process fault detection and diagnosis of multivariate processes.

Visual-Acoustic Penetration Recognition in Variable Polarity Plasma Arc Welding Process Using Hybrid Deep Learning Approach

Accurate characterization of keyhole pool behavior is important to improve the penetration recognition and quality detection during the variable polarity plasma arc welding (VPPAW) of aluminum alloy. However, the low-level hand-crafted visual/acoustic features are often incapable of sufficiently representing the dynamic characteristics of keyhole pool under complex welding conditions. In this paper, we developed an end-to-end visual-acoustic penetration recognition (VAPR) framework based on a hybrid convolutional neural network (CNN) and extreme learning machine (ELM), which consists of three consecutive phases: (1) visual-acoustic data preparation, (2) multi-features extraction, and (3) penetration classification. Specifically, we applied a flexible dual-sensor acquisition system for synchronously collecting/preprocessing the visual-acoustic signals and exploring the internal correlation between the visual-acoustic features and keyhole pool behavior as well as weld penetration status. Then we established two individual CNNs for learning high-level visual-acoustic features directly from the raw visual images and 2-D time-frequency spectrograms. With the visualization of CNN-based deep learning features, we also explained the physical meanings of visual-acoustic features. To further improve the prediction performance of CNN model, we employed the ELM model as a strong tool to classify penetration status. By comparing with other state-of-the-art methods, our hybrid CNN-ELM approach with visual-acoustic fusion has a superior performance in terms of the classifying accuracy. The VAPR framework in this paper will provide some guidance for post-process quality inspection and realize an adaptive control of VPPAW process.

Speech Emotion Recognition by Combining a Unified First-Order Attention Network With Data Balance

In the domain of speech emotion recognition (SER), generally there is an unbalanced data distribution of emotional samples in existing emotional datasets. Moreover, different fragment areas in an utterance contribute diversely to SER. To address these two issues, this paper proposes a new SER method by combining a unified first-order attention network with data balance. The proposed method firstly utilizes the strategy of data balance to augment and balance the training data. Then, a pre-trained convolutional neural network (CNN) model (i.e., VGGish) is fine-tuned on target emotional datasets to learn segment-level speech features from the extracted Log Mel-spectrograms. Next, the unified first-order attention mechanism, including different feature-pooling strategies such as sum, min, max, mean, and standard deviation (std), is embedded into the output of a bi-directional long short-term memory (Bi-LSTM) network. This is used for learning high-level discriminative segment-level features, and simultaneously aggregating the learned segment-level features into fixed-length utterance-level features for SER. Finally, based on utterance-level features, the softmax layer in a Bi-LSTM network is adopted to conduct final emotion classification task. Extensive experiments are implemented on three public datasets such as BAUM-1s, AFEW5.0, and CHEAVD2.0, demonstrate the advantage of the proposed method.

Deep online hierarchical dynamic unsupervised learning for pattern mining from utility usage data

While most non-intrusive load monitoring (NILM) work has focused on supervised algorithms, unsupervised approaches can be more interesting and practical. Specifically, they do not require labelled training data to be acquired from the individual appliances and can be deployed to operate on the measured aggregate data directly. We propose a fully unsupervised novel NILM framework based on Dynamic Bayesian hierarchical mixture model and Deep Belief network (DBN). The deep network learns, in unsupervised fashion, low-level generic appliance-specific features from the raw signals of the house utilities usage, then the hierarchical Bayesian model learns high-level features representing the consumption patterns of the residents captured by the correlations among the low-level features. The temporal ordering of the high-level features is captured by the Dynamic Bayesian Model. Using this architecture, we overcome the computational complexity that would occur if temporal modelling was directly applied to the raw data or even to the constructed features. The computational efficiency is crucial as our application involves massive data from different utilities usage. Moreover, we develop a novel online inference algorithm to cope with this big data. Finally, we propose different evaluation methods to analyse the results which show that our algorithm finds useful patterns.

ATTENTION BASED CONVOLUTIONAL NEURAL NETWORK FOR BUILDING EXTRACTION FROM VERY HIGH RESOLUTION REMOTE SENSING IMAGE

Abstract. Buildings are a major element in the formation of cities and are essential for urban mapping. The precise extraction of buildings from remote sensing data has become a significant topic and has received much attention in recent years. The recently developed convolutional neural networks have shown effective and superior performance to perform well on learning high-level and discriminative features in extracting buildings because of the outstanding feature learning and end-to-end pixel labelling abilities. However, it is difficult to use the features of different levels with a certain degree of importance that is appropriate to deep learning networks. To tackle this problem, a network based on U-Nets and the attention mechanism block was proposed. The network contains an encoder part and a decoder part and a spatial attention module. The special architecture presented in this article enhances the propagation of features and effectively utilizes the features at various levels to reduce errors. The other remarkable thing is that attention module blocks only lead to a minimal increase in model complexity. We effectively demonstrate an improvement of building extraction accuracy on challenging Potsdam and Vaihingen benchmark datasets. The results of this paper show that the proposed architecture improves building extraction in very high resolution remote sensing images compared to previous models.

CRIP: predicting circRNA-RBP-binding sites using a codon-based encoding and hybrid deep neural networks.

Circular RNAs (circRNAs), with their crucial roles in gene regulation and disease development, have become rising stars in the RNA world. To understand the regulatory function of circRNAs, many studies focus on the interactions between circRNAs and RNA-binding proteins (RBPs). Recently, the abundant CLIP-seq experimental data has enabled the large-scale identification and analysis of circRNA–RBP interactions, whereas, as far as we know, no computational tool based on machine learning has been proposed yet. We develop CRIP (CircRNAs Interact with Proteins) for the prediction of RBP-binding sites on circRNAs using RNA sequences alone. CRIP consists of a stacked codon-based encoding scheme and a hybrid deep learning architecture, in which a convolutional neural network (CNN) learns high-level abstract features and a recurrent neural network (RNN) learns long dependency in the sequences. We construct 37 data sets including sequence fragments of binding sites on circRNAs, and each set corresponds to an RBP. The experimental results show that the new encoding scheme is superior to the existing feature representation methods for RNA sequences, and the hybrid network outperforms conventional classifiers by a large margin, where both the CNN and RNN components contribute to the performance improvement.

Image quality assessment for advertising applications based on neural network

Image quality assessment (IQA) is a key technique in computer vision, which is widely applied in image classification, image aesthetic prediction. IQA plays an important role in advertising assessment system, which can recommend higher quality advertising for users. However, traditional algorithms cannot effectively predict advertising quality. In this paper, we propose an advertising assessment system using IQA algorithm based on neural networks. Specifically, we first incorporate both low-level features and high-level sematic features for image representation, where manifold learning algorithm is leveraged for high-level feature learning. Then, we leverage CNN based method for deep representation learning, which will be concatenated into deep feature vector. Finally, we leverage HMM model for learning image quality of advertising based on learned feature vector. Comprehensive experiments show the effectiveness of our proposed method.

Differential Diagnosis for Pancreatic Cysts in CT Scans Using Densely-Connected Convolutional Networks.

The lethal nature of pancreatic ductal adenocarcinoma (PDAC) calls for early differential diagnosis of pancreatic cysts, which are identified in up to 16% of normal subjects, and some of them may develop into PDAC. Pancreatic cysts have a large variation in size and shape, and the precise segmentation of them remains rather challenging, which restricts the computer-aided interpretation of CT images acquired for differential diagnosis. We propose a computer-aided framework for early differential diagnosis of pancreatic cysts without pre-segmenting the lesions using densely-connected convolutional networks (Dense-Net). The Dense-Net learns high-level features from whole abnormal pancreas and builds mappings between medical imaging appearance to different pathological types of pancreatic cysts. To enhance the clinical applicability, we integrate saliency maps in the framework to assist the physicians to understand the decision of the deep learning method. The test on a cohort of 206 patients with 4 pathologically confirmed subtypes of pancreatic cysts has achieved an overall accuracy of 72.8%, which is significantly higher than the baseline accuracy of 48.1%. The superior performance on this challenging dataset strongly supports the clinical potential of our developed method.

V-RSIR: A WEB-BASED TOOL AND BENCHMARK DATASET FOR REMOTE SENSING IMAGE RETRIEVAL

Abstract. Benchmark datasets play an important role in evaluating remote sensing image retrieval methods. Current benchmark datasets are mostly collected through the Google Map API or other desktop tools. However, the Google Map API requires the users to have programming skills and other collection tools are not publicly available, which may hinder the development of new benchmark datasets. This paper develops an open access web-based tool V-RSIR to help users generating new benchmark datasets with volunteers for remote sensing image retrieval. Using this tool, a new benchmark dataset V-RSIR that contains 38 classes with at least 1500 images per class is created by 32 volunteers. A handcrafted low-level feature method and a deep learning high-level feature method are used to test the dataset. The evaluation results are consistent with our perception. This shows that the tool can help users effectively creating benchmark datasets for RSIR.

Learning high-level features by fusing multi-view representation of MLS point clouds for 3D object recognition in road environments

Most existing 3D object recognition methods still suffer from low descriptiveness and weak robustness although remarkable progress has made in 3D computer vision. The major challenge lies in effectively mining high-level 3D shape features. This paper presents a high-level feature learning framework for 3D object recognition through fusing multiple 2D representations of point clouds. The framework has two key components: (1) three discriminative low-level 3D shape descriptors for obtaining multi-view 2D representation of 3D point clouds. These descriptors preserve both local and global spatial relationships of points from different perspectives and build a bridge between 3D point clouds and 2D Convolutional Neural Networks (CNN). (2) A two-stage fusion network, which consists of a deep feature learning module and two fusion modules, for extracting and fusing high-level features. The proposed method was tested on three datasets, one of which is Sydney Urban Objects dataset and the other two were acquired by a mobile laser scanning (MLS) system along urban roads. The results obtained from comprehensive experiments demonstrated that our method is superior to the state-of-the-art methods in descriptiveness, robustness and efficiency. Our method achieves high recognition rates of 94.6%, 93.1% and 74.9% on the above three datasets, respectively.

Pulmonary nodule segmentation with CT sample synthesis using adversarial networks.

Segmentation of pulmonary nodules is critical for the analysis of nodules and lung cancer diagnosis. We present a novel framework of segmentation for various types of nodules using convolutional neural networks (CNNs). The proposed framework is composed of two major parts. The first part is to increase the variety of samples and build a more balanced dataset. A conditional generative adversarial network (cGAN) is employed to produce synthetic CT images. Semantic labels are generated to impart spatial contextual knowledge to the network. Nine attribute scoring labels are combined as well to preserve nodule features. To refine the realism of synthesized samples, reconstruction error loss is introduced into cGAN. The second part is to train a nodule segmentation network on the extended dataset. We build a three-dimensional (3D) CNN model that exploits heterogeneous maps including edge maps and local binary pattern maps. The incorporation of these maps informs the model of texture patterns and boundary information of nodules, which assists high-level feature learning for segmentation. Residual unit, which learns to reduce residual error, is adopted to accelerate training and improve accuracy. Validation on LIDC-IDRI dataset demonstrates that the generated samples are realistic. The mean squared error and average cosine similarity between real and synthesized samples are and 0.9534, respectively. The Dice coefficient, positive predicted value, sensitivity, and accuracy are, respectively, 0.8483, 0.8895, 0.8511, and 0.9904 for the segmentation results. The proposed 3D CNN segmentation framework, based on the use of synthesized samples and multiple maps with residual learning, achieves more accurate nodule segmentation compared to existing state-of-the-art methods. The proposed CT image synthesis method can not only output samples close to real images but also allow for stochastic variation in image diversity.

Unsupervised Deep Noise Modeling for Hyperspectral Image Change Detection

Hyperspectral image (HSI) change detection plays an important role in remote sensing applications, and considerable research has been done focused on improving change detection performance. However, the high dimension of hyperspectral data makes it hard to extract discriminative features for hyperspectral processing tasks. Though deep convolutional neural networks (CNN) have superior capability in high-level semantic feature learning, it is difficult to employ CNN for change detection tasks. As a ground truth map is usually used for the evaluation of change detection algorithms, it cannot be directly used for supervised learning. In order to better extract discriminative CNN features, a novel noise modeling-based unsupervised fully convolutional network (FCN) framework is presented for HSI change detection in this paper. Specifically, the proposed method utilizes the change detection maps of existing unsupervised change detection methods to train the deep CNN, and then removes the noise during the end-to-end training process. The main contributions of this paper are threefold: (1) A new end-to-end FCN-based deep network architecture for HSI change detection is presented with powerful learning features; (2) An unsupervised noise modeling method is introduced for the robust training of the proposed deep network; (3) Experimental results on three datasets confirm the effectiveness of the proposed method.

Parallelized Convolutional Recurrent Neural Network With Spectral Features for Speech Emotion Recognition

Speech is the most effective way for people to exchange complex information. Recognition of emotional information contained in speech is one of the important challenges in the field of artificial intelligence. To better acquire emotional features in speech signals, a parallelized convolutional recurrent neural network (PCRN) with spectral features is proposed for speech emotion recognition. First, frame-level features are extracted from each utterance and, a long short-term memory is employed to learn these features frame by frame. At the same time, the deltas and delta-deltas of the log Mel-spectrogram are calculated and reconstructed into three channels (static, delta, and delta-delta); these 3-D features are learned by a convolutional neural network (CNN). Then, the two learned high-level features are fused and batch normalized. Finally, a SoftMax classifier is used to classify emotions. Our PCRN model simultaneously processes two different types of features in parallel to better learn the subtle changes in emotion. The experimental results on four public datasets show the superiority of our proposed method, which is better than the previous works.

V-RSIR: An Open Access Web-Based Image Annotation Tool for Remote Sensing Image Retrieval

Benchmark datasets play an important role in evaluating remote sensing image retrieval (RSIR) methods. At present, several small-scale benchmark datasets for RSIR are publicly available on the web and are mostly collected through the Google Map API or other desktop tools. Because the Google Map API requires the users to have programming skills and other collection tools are not publicly available, this may limit the possibility for a wide range of volunteers to participate in generating large-scale benchmark datasets. To address this challenge, we develop an open access web-based tool V-RSIR that allows volunteers to easily participate in generating new benchmark datasets for RSIR. This web-based tool not only facilitates the remote sensing image label and cropping, but also provides image editing, review, quantity statistics, spatial distribution, sharing, and so on. To validate this tool, we recruit 32 volunteers to label and crop remote sensing images by using the tool. Finally, a new benchmark dataset that contains 38 classes with at least 1500 images per class is created. Then, the new dataset is validated by five handcrafted low-level feature methods and four deep learning high-level feature methods. The experimental results show that the handcrafted low-level feature methods perform worse than the deep learning methods, in which the precision at top 5 can achieve 94%. The evaluation results are consistent with our theoretical understanding and experimental results on the PatternNet dataset. This indicates that our web-based tool can help users generating valid benchmark datasets with volunteers for the RSIR.

A Framework Integrating Heterogeneous Databases for the Completion of Gene Networks

Molecular networks embraced diverse biological and functional associations between genes and gene products, which are conducive for identifying novel genes and pathways of a specific disease phenotype. Although great progress has been achieved in high-throughput interactome mapping, the associations among genes still incomplete which caused the sparsity of Gene Networks (GNs). Here, we proposed a network-based framework, termed NIHO, for optimizing and completing GNs by integrating six genome networks: STRING, ConsenusPathDB, HumanNet, GeneMANIA, GIANT and BioGRID. NIHO learns high-level features of genes from the heterogeneous networks by an end-to-end way that consisted of neural network and matrix completion. Then, the learned low-dimensional representations are used to calculate the geometric proximity of genes in the projected space. Finally, NIHO infers the interactions among genes by analyzing the proximity scores and adds those interactions that originally not existed into GNs. The experimental results showed that the capability of GNs to recover disease gene sets get significantly improved after processed by NIHO. In addition, we not only examined the proportion of the added interactions but also observed the performance of NIHO can be promoted when the number of heterogeneous networks is increased.

Dynamic Multicontext Segmentation of Remote Sensing Images Based on Convolutional Networks

Semantic segmentation requires methods capable of learning high-level features while dealing with large volume of data. Towards such goal, Convolutional Networks can learn specific and adaptable features based on the data. However, these networks are not capable of processing a whole remote sensing image, given its huge size. To overcome such limitation, the image is processed using fixed size patches. The definition of the input patch size is usually performed empirically (evaluating several sizes) or imposed (by network constraint). Both strategies suffer from drawbacks and could not lead to the best patch size. To alleviate this problem, several works exploited multi-context information by combining networks or layers. This process increases the number of parameters resulting in a more difficult model to train. In this work, we propose a novel technique to perform semantic segmentation of remote sensing images that exploits a multi-context paradigm without increasing the number of parameters while defining, in training time, the best patch size. The main idea is to train a dilated network with distinct patch sizes, allowing it to capture multi-context characteristics from heterogeneous contexts. While processing these varying patches, the network provides a score for each patch size, helping in the definition of the best size for the current scenario. A systematic evaluation of the proposed algorithm is conducted using four high-resolution remote sensing datasets with very distinct properties. Our results show that the proposed algorithm provides improvements in pixelwise classification accuracy when compared to state-of-the-art methods.

Deep Blur Mapping: Exploiting High-Level Semantics by Deep Neural Networks.

The human visual system excels at detecting local blur of visual images, but the underlying mechanism is not well understood. Traditional views of blur such as reduction in energy at high frequencies and loss of phase coherence at localized features have fundamental limitations. For example, they cannot well discriminate flat regions from blurred ones. Here we propose that high-level semantic information is critical in successfully identifying local blur. Therefore, we resort to deep neural networks that are proficient at learning high-level features and propose the first end-to-end local blur mapping algorithm based on a fully convolutional network. By analyzing various architectures with different depths and design philosophies, we empirically show that high-level features of deeper layers play a more important role than low-level features of shallower layers in resolving challenging ambiguities for this task. We test the proposed method on a standard blur detection benchmark and demonstrate that it significantly advances the state-of-the-art (ODS F-score of 0.853). Furthermore, we explore the use of the generated blur maps in three applications, including blur region segmentation, blur degree estimation, and blur magnification.

Saliency Detection and Deep Learning-Based Wildfire Identification in UAV Imagery.

An unmanned aerial vehicle (UAV) equipped with global positioning systems (GPS) can provide direct georeferenced imagery, mapping an area with high resolution. So far, the major difficulty in wildfire image classification is the lack of unified identification marks, the fire features of color, shape, texture (smoke, flame, or both) and background can vary significantly from one scene to another. Deep learning (e.g., DCNN for Deep Convolutional Neural Network) is very effective in high-level feature learning, however, a substantial amount of training images dataset is obligatory in optimizing its weights value and coefficients. In this work, we proposed a new saliency detection algorithm for fast location and segmentation of core fire area in aerial images. As the proposed method can effectively avoid feature loss caused by direct resizing; it is used in data augmentation and formation of a standard fire image dataset ‘UAV_Fire’. A 15-layered self-learning DCNN architecture named ‘Fire_Net’ is then presented as a self-learning fire feature exactor and classifier. We evaluated different architectures and several key parameters (drop out ratio, batch size, etc.) of the DCNN model regarding its validation accuracy. The proposed architecture outperformed previous methods by achieving an overall accuracy of 98%. Furthermore, ‘Fire_Net’ guarantied an average processing speed of 41.5 ms per image for real-time wildfire inspection. To demonstrate its practical utility, Fire_Net is tested on 40 sampled images in wildfire news reports and all of them have been accurately identified.

Face Recognition Based on CSGF(2D)2PCANet

Face recognition has a great potential to play an important role in computer vision field. However, the majority of face recognition methods are based on the low-level features, which may not yield good results. Inspired by a simple deep learning model principal component analysis network (PCANet), we propose a novel deep learning network called circular symmetrical Gabor filter (2D)2PCA neural networks [CSGF(2D)2PCANet]. Previous models used in face recognition have three major issues of data redundancy, computation time, and no rotation invariance. We introduce the CSGF to address these issues. Two-directional 2-D PCA [(2D)2PCA] is used in feature extraction stage. Binary hashing, blockwise histograms, and linear SVM are used for the output stage. The proposed CSGF (2D)2PCANet learns high-level features and provides more recognition information during the training phase, which may result in a higher recognition rate when testing the sample. We tested the proposed method on XM2VTS, ORL, AR, Extend Yale B, and LFW databases. Test results show that the CSGF (2D)2PCANet is more robust to the variation of occlusion, illumination, pose, noise, and expression, which is a promising method in face recognition.