Unsupervised Feature Learning Research Articles

Unsupervised Feature Learning With Winner-Takes-All Based STDP.

We present a novel strategy for unsupervised feature learning in image applications inspired by the Spike-Timing-Dependent-Plasticity (STDP) biological learning rule. We show equivalence between rank order coding Leaky-Integrate-and-Fire neurons and ReLU artificial neurons when applied to non-temporal data. We apply this to images using rank-order coding, which allows us to perform a full network simulation with a single feed-forward pass using GPU hardware. Next we introduce a binary STDP learning rule compatible with training on batches of images. Two mechanisms to stabilize the training are also presented : a Winner-Takes-All (WTA) framework which selects the most relevant patches to learn from along the spatial dimensions, and a simple feature-wise normalization as homeostatic process. This learning process allows us to train multi-layer architectures of convolutional sparse features. We apply our method to extract features from the MNIST, ETH80, CIFAR-10, and STL-10 datasets and show that these features are relevant for classification. We finally compare these results with several other state of the art unsupervised learning methods.

Research on weeds identification based on K-means feature learning

This paper aims to overcome the unstable identification results and weak generalization ability in feature extraction based on manual design to realize the automatic weeds identification. On the basis of unsupervised feature learning identification model, K-means clustering algorithm after data preprocessing is used to realize feature learning and construct feature dictionary. Then this feature dictionary is used to extract features from labeled data and train the classification model to realize the automatic weeds identification. In this process, this paper focuses on the effect of parameters such as the clustering number to identification accuracy under single-layer network structure, and the identification accuracy between the single-layer and the two-layer network structure was compared and analyzed. Experimental results show that identification rate can be improved by increasing the network levels, as well as fine-tuning the parameters under the premise of selecting reasonable parameters.

Deep Spatiality: Unsupervised Learning of Spatially-Enhanced Global and Local 3D Features by Deep Neural Network with Coupled Softmax.

The discriminability of Bag-of-Words representations can be increased via encoding the spatial relationship among virtual words on 3D shapes. However, this encoding task involves several issues, including arbitrary mesh resolutions, irregular vertex topology, orientation ambiguity on 3D surface, invariance to rigid and non-rigid shape transformations. To address these issues, a novel unsupervised spatial learning framework based on deep neural network, deep spatiality (DS), is proposed. Specifically, DS employs two novel components: spatial context extractor and deep context learner. Spatial context extractor extracts the spatial relationship among virtual words in a local region into a raw spatial representation. Along a consistent circular direction, a directed circular graph is constructed to encode relative positions between pairwise virtual words in each face ring into a relative spatial matrix. By decomposing each relative spatial matrix using SVD, the raw spatial representation is formed, from which deep context learner conducts unsupervised learning of global and local features. Deep context learner is a deep neural network with a novel model structure to adapt the proposed coupled softmax layer, which encodes not only the discriminative information among local regions but also the one among global shapes. Experimental results show that DS outperforms state-of-the-art methods.

Spatially Coherent Feature Learning for Pose-Invariant Facial Expression Recognition

Feature learning has enjoyed much attention and achieved good performance in recent studies of image processing. Unlike the required training conditions often assumed there, far less labeled data is available for training emotion classification systems. In addition, current feature learning is typically performed on an entire face image without considering the dependency between features. These approaches ignore the fact that faces are structured and the neighboring features are dependent. Thus, the learned features lack the power to describe visually coherent facial images. Our method is therefore designed with the goal of simplifying the problem domain by removing expression-irrelevant factors from the input images, with a key region-based mechanism, which is an effort to reduce the amount of data required to effectively train the feature-learning methods. Meanwhile, we can construct geometric constraints between the key regions and its detected positions. To this end, we introduce a Spatially Coherent featurelearning method for Pose-invariant Facial Expression Recognition (SC-PFER). In our model, we first perform face frontalization through a 3D pose-normalization technique, which could normalize poses while preserving the identity information through synthesizing frontal faces for facial images with arbitrary views. Subsequently, we select a sequence of key regions around 51 key points in the synthetic frontal face images for efficient unsupervised feature learning. Finally, we introduce a linkage structure over the learning-based features and the corresponding geometry information of each key region to encode the dependencies of the regions. Our method, on the whole, does not require training multiple models for each specific pose and avoids separating training and parameter tuning for each pose. The proposed framework has been evaluated on two benchmark databases, BU-3DFE and SFEW, for pose-invariant Facial Expression Recognition (FER). The experimental results demonstrate that our algorithm outperforms current state-of-the-art FER methods. Specifically, our model achieves an improvement of 1.72% and 1.11% FER accuracy, on average, on BU-3DFE and SFEW, respectively.

Unsupervised geographically discriminative feature learning for landmark tagging

Recently, a large number of geo-tagged landmark images have been uploaded through various social media services. Usually, these geo-tagged images are annotated by users with GPS and tags related to the landmarks where they are taken. Landmark tagging aims to automatically annotate an image with the tags to describe the landmark where the image is taken. It has been observed that the images and tags show strong correlation with the geographical locations. The widely used assumption by many existing tagging methods is that images are independently and identically distributed is not effective to capture the geographical correlation. In this paper, we study the novel problem of utilizing the geographical correlation among images and landmarks for better tagging landmark images. In particular, we propose an unsupervised feature learning approach to learn the geographically discriminative features across geographical locations, by integrating latent space learning and geographically structural analysis (LSGSA) into a joint model. A latent space learning model is proposed to effectively fuse the heterogeneous features of visual content and tags. Meanwhile, the geographical structure analysis and group sparsity are applied to learn the geographically discriminative features. Then, a geo-guided sparse reconstruction method is proposed to tag images by utilizing the discriminative information of features, in which the landmark-specific tags are boosted by a weighting method. Experiments on the real-world datasets demonstrate the superiority of our approach.

QoE Multi-Stage Machine Learning for Dynamic Video Streaming

The rapid growth of video traffic in cellular networks is a crucial issue to be addressed by mobile operators. An emerging and promising trend in this regard is the development of solutions that aim at maximizing the quality of experience (QoE) of the end users. However, predicting the QoE perceived by the users in different conditions remains a major challenge. In this paper, we propose a machine learning approach to support QoE-based video admission control and resource management algorithms. More specifically, we develop a multi-stage learning system that combines the unsupervised learning of video features from the size of H.264-encoded video frames with a supervised classifier trained to automatically extract the quality-rate characteristics of unknown video sequences. This QoE characterization is then used to manage simultaneous video transmissions through a shared channel in order to guarantee a minimum quality level delivered to the final users. Simulation results show that the proposed video admission control and resource management algorithms, which are based on learning-based QoE classification of video sequences, outperform standard content-agnostic strategies.

Unsupervised Feature Learning with Single Layer ICANet for Face Recognition

Compared to supervised learning, unsupervised learning allows systems to learn consistent patterns from cheap and abundant unlabeled data, without the need for manual annotation. In this paper, we present a novel unsupervised feature learning method with single layer (SL) network based independent component analysis (ICA) filters called SL-ICANet, with the goal of achieving compact and robust facial feature representation. Our contributions are twofold: (i) We developed a single-layer convolutional network for unsupervised learning wherein the trainable kernels are replaced by ICA filters. (ii) We extended our SL-ICANet to use multi-scale information for better feature learning. Extensive experiments on two popular face recognition benchmarks, namely, labeled faces in the wild and facial recognition technology show that the proposed method might serve as a simple but highly competitive baseline for face recognition.

When Low Rank Representation Based Hyperspectral Imagery Classification Meets Segmented Stacked Denoising Auto-Encoder Based Spatial-Spectral Feature

When confronted with limited labelled samples, most studies adopt an unsupervised feature learning scheme and incorporate the extracted features into a traditional classifier (e.g., support vector machine, SVM) to deal with hyperspectral imagery classification. However, these methods have limitations in generalizing well in challenging cases due to the limited representative capacity of the shallow feature learning model, as well as the insufficient robustness of the classifier which only depends on the supervision of labelled samples. To address these two problems simultaneously, we present an effective low-rank representation-based classification framework for hyperspectral imagery. In particular, a novel unsupervised segmented stacked denoising auto-encoder-based feature learning model is proposed to depict the spatial-spectral characteristics of each pixel in the imagery with deep hierarchical structure. With the extracted features, a low-rank representation based robust classifier is then developed which takes advantage of both the supervision provided by labelled samples and unsupervised correlation (e.g., intra-class similarity and inter-class dissimilarity, etc.) among those unlabelled samples. Both the deep unsupervised feature learning and the robust classifier benefit, improving the classification accuracy with limited labelled samples. Extensive experiments on hyperspectral imagery classification demonstrate the effectiveness of the proposed framework.

Convolutional adaptive denoising autoencoders for hierarchical feature extraction

Convolutional neural networks (CNNs) are typical structures for deep learning and are widely used in image recognition and classification. However, the random initialization strategy tends to become stuck at local plateaus or even diverge, which results in rather unstable and ineffective solutions in real applications. To address this limitation, we propose a hybrid deep learning CNN-AdapDAE model, which applies the features learned by the AdapDAE algorithm to initialize CNN filters and then train the improved CNN for classification tasks. In this model, AdapDAE is proposed as a CNN pre-training procedure, which adaptively obtains the noise level based on the principle of annealing, by starting with a high level of noise and lowering it as the training progresses. Thus, the features learned by AdapDAE include a combination of features at different levels of granularity. Extensive experimental results on STL-10, CIFAR-10, andMNIST datasets demonstrate that the proposed algorithm performs favorably compared to CNN (random filters), CNNAE (pre-training filters by autoencoder), and a few other unsupervised feature learning methods.

A Deep Learning Approach to Network Intrusion Detection

Network intrusion detection systems (NIDSs) play a crucial role in defending computer networks. However, there are concerns regarding the feasibility and sustainability of current approaches when faced with the demands of modern networks. More specifically, these concerns relate to the increasing levels of required human interaction and the decreasing levels of detection accuracy. This paper presents a novel deep learning technique for intrusion detection, which addresses these concerns. We detail our proposed nonsymmetric deep autoencoder (NDAE) for unsupervised feature learning. Furthermore, we also propose our novel deep learning classification model constructed using stacked NDAEs. Our proposed classifier has been implemented in graphics processing unit (GPU)-enabled TensorFlow and evaluated using the benchmark KDD Cup ’99 and NSL-KDD datasets. Promising results have been obtained from our model thus far, demonstrating improvements over existing approaches and the strong potential for use in modern NIDSs.

Deep diagnostics and prognostics: An integrated hierarchical learning framework in PHM applications

Prognostics and Health Management (PHM) is an integrated technique for improving the availability and efficiency of high-value industry equipment and reducing the maintenance cost. One of the most challenging problems in PHM is how to effectively process the raw monitoring signal into the information-rich features that are readable enough for PHM modeling. In this paper, we propose an integrated hierarchical learning framework, which is capable to perform the unsupervised feature learning, diagnostics and prognostics modeling together. The proposed method is based on Auto-Encoders (trained by considering the L1-norm penalty) and Extreme Learning Machines (trained by considering the L2-norm penalty). The proposed method is applied on two different case studies considering the diagnostics of motor bearings and prognostics of turbofan engines, also the performances are compared with other commonly applied PHM approaches and machine learning tools. The obtained results demonstrate the superiority of the proposed method, especially the ability of extracting the relevant features from the non-informative and noisy signals and maintaining their efficiencies.

Role of zero synapses in unsupervised feature learning

Synapses in real neural circuits can take discrete values including zero (silent or potential) synapses. The computational role of zero synapses in unsupervised feature learning of unlabeled noisy data is still unclear, thus it is important to understand how the sparseness of synaptic activity is shaped during learning and its relationship with receptive field formation. Here, we formulate this kind of sparse feature learning by a statistical mechanics approach. We find that learning decreases the fraction of zero synapses, and when the fraction decreases rapidly around a critical data size, an intrinsically structured receptive field starts to develop. Further increasing the data size refines the receptive field, while a very small fraction of zero synapses remain to act as contour detectors. This phenomenon is discovered not only in learning a handwritten digits dataset, but also in learning retinal neural activity measured in a natural-movie-stimuli experiment.

Unsupervised Spectral–Spatial Feature Learning via Deep Residual Conv–Deconv Network for Hyperspectral Image Classification

Supervised approaches classify input data using a set of representative samples for each class, known as training samples . The collection of such samples is expensive and time demanding. Hence, unsupervised feature learning, which has a quick access to arbitrary amounts of unlabeled data, is conceptually of high interest. In this paper, we propose a novel network architecture, fully Conv–Deconv network, for unsupervised spectral–spatial feature learning of hyperspectral images, which is able to be trained in an end-to-end manner. Specifically, our network is based on the so-called encoder–decoder paradigm, i.e., the input 3-D hyperspectral patch is first transformed into a typically lower dimensional space via a convolutional subnetwork (encoder), and then expanded to reproduce the initial data by a deconvolutional subnetwork (decoder). However, during the experiment, we found that such a network is not easy to be optimized. To address this problem, we refine the proposed network architecture by incorporating: 1) residual learning and 2) a new unpooling operation that can use memorized max-pooling indexes. Moreover, to understand the “black box,” we make an in-depth study of the learned feature maps in the experimental analysis. A very interesting discovery is that some specific “neurons” in the first residual block of the proposed network own good description power for semantic visual patterns in the object level, which provide an opportunity to achieve “free” object detection. This paper, for the first time in the remote sensing community, proposes an end-to-end fully Conv–Deconv network for unsupervised spectral–spatial feature learning. Moreover, this paper also introduces an in-depth investigation of learned features. Experimental results on two widely used hyperspectral data, Indian Pines and Pavia University, demonstrate competitive performance obtained by the proposed methodology compared with other studied approaches.

An intelligent health monitoring method for processing data collected from the sensor network of structure

Rapid detection of damages in civil engineering structures, in order to assess their possible disorders and as a result produce competent decision making, are crucial to ensure their health and ultimately enhance the level of public safety. In traditional intelligent health monitoring methods, the features are manually extracted depending on prior knowledge and diagnostic expertise. Inspired by the idea of unsupervised feature learning that uses artificial intelligence techniques to learn features from raw data, a two-stage learning method is proposed here for intelligent health monitoring of civil engineering structures. In the first stage, Nystrom method is used for automatic feature extraction from structural vibration signals. In the second stage, Moving Kernel Principal Component Analysis (MKPCA) is employed to classify the health conditions based on the extracted features. In this paper, KPCA has been implemented in a new form as Moving KPCA for effectively segmenting large data and for determining the changes, as data are continuously collected. Numerical results revealed that the proposed health monitoring system has a satisfactory performance for detecting the damage scenarios of a three-story frame aluminum structure. Furthermore, the enhanced version of KPCA methods exhibited a significant improvement in sensitivity, accuracy, and effectiveness over conventional methods.

Deep Convolution Neural Network and Autoencoders-Based Unsupervised Feature Learning of EEG Signals

Epilepsy is a health problem that seriously affects the quality of humans for many years. Therefore, it is important to accurately analyze and recognize epilepsy based on EEG signals, and for a long time, researchers have attempted to extract new features from the signals for epilepsy recognition. However, it is very difficult to select useful features from a large number of them in this diagnostic application. As the development of artificial intelligence progresses, unsupervised feature learning based on the deep learning model can obtain features that can better describe identified objects from unlabeled data. In this paper, the deep convolution network and autoencoders-based model, named as AE-CDNN, is constructed in order to perform unsupervised feature learning from EEG in epilepsy. We extract features by AE-CDNN model and classify the features based on two public EEG data sets. Experimental results showed that the classification results of features obtained by AE-CDNN are more optimal than features obtained by principal component analysis and sparse random projection. Using several common classifiers to classify features obtained by AE-CDNN model results in high accuracy and not inferior to the research results from most recent studies. The results also showed that the features of AE-CDNN model are clear, effective, and easy to learn. These features can speed up the convergence and reduce the training times of classifiers. Therefore, the AE-CDNN model can be effectively applied to feature extraction of EEG in epilepsy.

A Homotopy Iterative Hard Thresholding Algorithm With Extreme Learning Machine for Scene Recognition

Vision-based scene recognition aims to find a semantic explanation of a scene image. First, how to learn a sparse feature representation by mapping each cluster of patches into few dimensions is one challenging issue. Sparse coding techniques with $l_{1}$ norm regularization are widely used to learn sparse features. However, $l_{1}$ norm regularization may not achieve enough sparsity. Furthermore, the initial value has a great influence on the sparsity of codes during the iterative optimization process. Second, how to learn a classifier in a supervised manner in order to improve the generalization is another issue. This paper therefore proposes a scene recognition method and it mainly includes two processes. One is the homotopy iterative hard thresholding (HIHT) algorithm that encodes the sparse representations of local patches by incorporating an $l_{0}$ norm regularization. Furthermore, a homotopy continuity strategy is used to improve the sparsity of feature codes by adaptively tuning the regularization factor from large to small values and using the sparse solution of last iteration as the warm start of the next iteration. The other is the extreme learning machine (ELM)-based classifier. Experimental results in 15-class scene data set have shown that the HIHT algorithm outperforms other unsupervised sparse feature learning algorithms in terms of sparsity and entropy. Meanwhile, the ELM-based scene recognition method outperforms the other state-of-the art methods in terms of recognition accuracy.

Deep Learning Technology for Predicting Solar Flares from (Geostationary Operational Environmental Satellite) Data

Solar activity, particularly solar flares can have significant detrimental effects on both space-borne and grounds based systems and industries leading to subsequent impacts on our lives. As a consequence, there is much current interest in creating systems which can make accurate solar flare predictions. This paper aims to develop a novel framework to predict solar flares by making use of the Geostationary Operational Environmental Satellite (GOES) X-ray flux 1-minute time series data. This data is fed to three integrated neural networks to deliver these predictions. The first neural network (NN) is used to convert GOES X-ray flux 1-minute data to Markov Transition Field (MTF) images. The second neural network uses an unsupervised feature learning algorithm to learn the MTF image features. The third neural network uses both the learned features and the MTF images, which are then processed using a Deep Convolutional Neural Network to generate the flares predictions. To the best of our knowledge, this work is the first flare prediction system that is based entirely on the analysis of pre-flare GOES X-ray flux data. The results are evaluated using several performance measurement criteria that are presented in this paper.

Unsupervised Learning of 3-D Local Features From Raw Voxels Based on a Novel Permutation Voxelization Strategy.

Effective 3-D local features are significant elements for 3-D shape analysis. Existing hand-crafted 3-D local descriptors are effective but usually involve intensive human intervention and prior knowledge, which burdens the subsequent processing procedures. An alternative resorts to the unsupervised learning of features from raw 3-D representations via popular deep learning models. However, this alternative suffers from several significant unresolved issues, such as irregular vertex topology, arbitrary mesh resolution, orientation ambiguity on the 3-D surface, and rigid and slightly nonrigid transformation invariance. To tackle these issues, we propose an unsupervised 3-D local feature learning framework based on a novel permutation voxelization strategy to learn high-level and hierarchical 3-D local features from raw 3-D voxels. Specifically, the proposed strategy first applies a novel voxelization which discretizes each 3-D local region with irregular vertex topology and arbitrary mesh resolution into regular voxels, and then, a novel permutation is applied to permute the voxels to simultaneously eliminate the effect of rotation transformation and orientation ambiguity on the surface. Based on the proposed strategy, the permuted voxels can fully encode the geometry and structure of each local region in regular, sparse, and binary vectors. These voxel vectors are highly suitable for the learning of hierarchical common surface patterns by stacked sparse autoencoder with hierarchical abstraction and sparse constraint. Experiments are conducted on three aspects for evaluating the learned local features: 1) global shape retrieval; 2) partial shape retrieval; and 3) shape correspondence. The experimental results show that the learned local features outperform the other state-of-the-art 3-D shape descriptors.

Predicting and Grouping Digitized Paintings by Style using Unsupervised Feature Learning.

To create an system to aid in the analysis of art history by classifying and grouping digitized paintings based on stylistic features automatically learned without prior knowledge. 6,776 digitized paintings from 8 different artistic styles (Art Nouveau, Baroque, Expressionism, Impressionism, Realism, Romanticism, Renaissance, and Post-Impressionism) were utilized to classify (predict) and cluster (group) paintings according to style. The method of unsupervised feature learning with K-means (UFLK), inspired by deep learning, was utilized to extract features from the paintings. These features were then used in: (1) a support vector machine algorithm to classify the style of new test paintings based on a training set of paintings having known style labels; and (2) a spectral clustering algorithm to group the paintings into distinct style groups (anonymously, without employing any known style labels). Classification performance was determined by accuracy and F-score. Clustering performance was determined by: 1) the ability to recover the original stylistic groupings (using a cost analysis of all possible combinations of 8 group label assignments); 2) F-score; and 3) a reliability analysis. The latter analysis used two novel ways to determine the distribution of the null-hypothesis: (a) a uniform distribution projected onto the principal components of the original data; and (b) a randomized, weighted adjacency matrix. The ability to gain insights into art was tested by a semantic analysis of the clustering results. For this purpose, we represented the featural characteristics of each painting by an N-dimensional feature vector, and plotted the distance between vector endpoints (i.e., similarity between paintings). Then, we color-coded the endpoints with the assigned lowest-cost style labels. The scatterplot was visually inspected for separation of the paintings, where the amount of separation between color clusters provides semantic information on the interrelatedness between styles. The UFLK-extracted features resembled the edges/lines/colors in the paintings. For feature-based classification of paintings, the macro-averaged F-score was 0.469. Classification accuracy and F-score were similar/higher compared to other classification methods using more complex feature learning models (e.g., convolutional neural networks, a supervised algorithm). The clustering via UFLK-extracted features yielded 8 unlabeled style groupings. In 6 of 8 clusters, the most common true painting style matched the cluster style assigned by cost analysis. The clustering had an F-score of 0.212. (There are no comparison methods for clustering paintings.) For the semantic analysis, the featural characteristics of Baroque and Art Nouveau were found to be similar, indicating a relationship between these styles. The UFLK method can extract features from digitized paintings. We were able to extract characteristics of art without any prior information about the nature of the features or the stylistic designation of the paintings. The methods herein may provide art researchers with the latest computational techniques for the documentation, interpretation, and forensics of art. The tools could assist the preservation of culturally sensitive works of art for future generations, and provide new insights into works of art and the artists who created them.

Enhancing interpretability of automatically extracted machine learning features: application to a RBM-Random Forest system on brain lesion segmentation.

Machine learning systems are achieving better performances at the cost of becoming increasingly complex. However, because of that, they become less interpretable, which may cause some distrust by the end-user of the system. This is especially important as these systems are pervasively being introduced to critical domains, such as the medical field. Representation Learning techniques are general methods for automatic feature computation. Nevertheless, these techniques are regarded as uninterpretable "black boxes". In this paper, we propose a methodology to enhance the interpretability of automatically extracted machine learning features. The proposed system is composed of a Restricted Boltzmann Machine for unsupervised feature learning, and a Random Forest classifier, which are combined to jointly consider existing correlations between imaging data, features, and target variables. We define two levels of interpretation: global and local. The former is devoted to understanding if the system learned the relevant relations in the data correctly, while the later is focused on predictions performed on a voxel- and patient-level. In addition, we propose a novel feature importance strategy that considers both imaging data and target variables, and we demonstrate the ability of the approach to leverage the interpretability of the obtained representation for the task at hand. We evaluated the proposed methodology in brain tumor segmentation and penumbra estimation in ischemic stroke lesions. We show the ability of the proposed methodology to unveil information regarding relationships between imaging modalities and extracted features and their usefulness for the task at hand. In both clinical scenarios, we demonstrate that the proposed methodology enhances the interpretability of automatically learned features, highlighting specific learning patterns that resemble how an expert extracts relevant data from medical images.