Abundant Unlabeled Data Research Articles

Divide and conquer anomaly detection: A case study predicting defective engines

The goal of this paper is to test if data partitioning based on domain knowledge improves the performance of unsupervised anomaly detection algorithms for the detection of faulty internal combustion engines. We test three common anomaly detection algorithms to predict defective engines at the end of an assembly line, both with and without data partitioning. The algorithms are trained on high-dimensional vibration data. To evaluate and compare the detection performance of partitioned and unpartitioned approaches, we use a labeled collection of known anomalies. Using domain knowledge to partition the data improves the anomaly detection performance of all tested algorithms. A divide and conquer strategy based on data partitioning, thus, appears to be a viable anomaly detection approach in cases where abundant unlabeled data is available.

Improved well-log classification using semisupervised label propagation and self-training, with comparisons to popular supervised algorithms

Machine-learning techniques allow geoscientists to extract meaningful information from data in an automated fashion, and they are also an efficient alternative to traditional manual interpretation methods. Many geophysical problems have an abundance of unlabeled data and a paucity of labeled data, and the lithology classification of wireline data reflects this situation. Training supervised algorithms on small labeled data sets can lead to overtraining, and subsequent predictions for the numerous unlabeled data may be unstable. However, semisupervised algorithms are designed for classification problems with limited amounts of labeled data, and they are theoretically able to achieve better accuracies than supervised algorithms in these situations. We explore this hypothesis by applying two semisupervised techniques, label propagation (LP) and self-training, to a well-log data set and compare their performance to three popular supervised algorithms. LP is an established method, but our self-training method is a unique adaptation of existing implementations. The well-log data were made public through an SEG competition held in 2016. We simulate a semisupervised scenario with these data by assuming that only one of the 10 wells has labels (i.e., core samples), and our objective is to predict the labels for the remaining nine wells. We generate results from these data in two stages. The first stage is applying all the algorithms in question to the data as is (i.e., the global data), and the results from this motivate the second stage, which is applying all algorithms to the data when they are decomposed into two separate data sets. Overall, our findings suggest that LP does not outperform the supervised methods, but our self-training method coupled with LP can outperform the supervised methods by a notable margin if the assumptions of LP are met.

Exploring High-Order Correlations for Industry Anomaly Detection

Anomaly detection aims to find the outlier noise data, which has attracted much attention in recent years. However, in most industrial cases, we can only obtain very few labeled anomalous data with abundant unlabeled data, which makes anomaly detection intractable. To tackle this issue, in this paper we propose a vertex-weighted hypergraph learning method for anomaly detection. In this method, the correlation among data is formulated in a hypergraph structure, where each vertex denotes one sample, and the connections (hyperedges) on the hypergraph represent the relation among samples in feature space. We introduce vertex weights to investigate the importance of different samples, where each vertex is initialized with a weight corresponding to its similarity score and isolation score. Then, we learn the label projection matrix for anomaly detection and optimize vertex weights simultaneously. In this way, the vertices with high weights play an important role during the learning process. We have evaluated our method on industry anomaly detection dataset, outlier detection dataset, and software defect prediction dataset, and experimental results show that our method achieves better performance when compared with other state-of-the-art methods.

Adaptive Hypergraph Embedded Semi-Supervised Multi-Label Image Annotation

Multilabel image annotation attracts a lot of research interest due to its practicability in multimedia and computer vision fields, while the need for a large amount of labeled training data to achieve promising performance makes it a challenging task. Fortunately, unlabeled and relevant data are widely available and these data can be used to serve the annotation task. To this end, we propose a novel adaptive hypergraph learning (AHL) method for multilabel image annotation in a semisupervised way, in which both the limited labeled data and abundant unlabeled data are utilized to facilitate the annotation performance. In detail, we seek a multilabel propagation scheme by learning a hypergraph which is used to preserve the local geometric structures of data in a high-order manner. Meanwhile, a feature projection is integrated into AHL to obtain a latent feature space where unlabeled instances can be effectively and robustly assigned with multiple labels. Experiments on six widely used image datasets are conducted to evaluate our model and the results demonstrate that the proposed AHL outperforms other state-of-the-art semisupervised methods.

Urban scene semantic segmentation with insufficient labeled data

Semantic segmentation of urban scenes is an enabling factor for a wide range of applications. With the development of deep learning in recent years, semantic segmentation tasks using high-capacity models have achieved considerable successes on large datasets. However, the pixel-level annotation process, especially for urban scene images with various objects, is tedious and labor intensive. Meanwhile, the scale of the unlabeled data, which is currently easy to collect, is often much larger than labeled data. Thus, using the abundant unlabeled data to make up the loss of the segmentation model from insufficient labeled data is of great interest. In this paper, we propose a semi-supervised method based on reinforcement learning to capture the contextual information from the unlabeled data to improve the model trained on the small scale labeled data. Both quantitative and qualitative experiments have shown the effectiveness of the proposed method.

Fault Classification in High-Dimensional Complex Processes Using Semi-Supervised Deep Convolutional Generative Models

In complex industrial processes, process fault detection and classification constitute an important task for reducing production costs and improving product quality. Most existing methods for fault classification assume that sufficient labeled data are available for training. However, label acquisition is costly and laborious in practice, whereas abundant unlabeled data are often available. To make effective use of a large amount of unlabeled data for fault classification, we propose in this article a new approach using semi-supervised deep generative models, allowing the complex relationship between high-dimensional process data and the process status to be modeled. In particular, to consider the temporal correlation and intervariable correlation in multivariate time series process data collected from multiple sensors, we propose two semi-supervised deep generative models incorporating convolutional neural networks. The proposed models are assessed on data from the Tennessee Eastman benchmark process. The results demonstrate the superior performances of the proposed models compared with competing methods.

Unsupervised Feature Selection by Pareto Optimization

Dimensionality reduction is often employed to deal with the data with a huge number of features, which can be generally divided into two categories: feature transformation and feature selection. Due to the interpretability, the efficiency during inference and the abundance of unlabeled data, unsupervised feature selection has attracted much attention. In this paper, we consider its natural formulation, column subset selection (CSS), which is to minimize the reconstruction error of a data matrix by selecting a subset of features. We propose an anytime randomized iterative approach POCSS, which minimizes the reconstruction error and the number of selected features simultaneously. Its approximation guarantee is well bounded. Empirical results exhibit the superior performance of POCSS over the state-of-the-art algorithms.

Unsupervised Learning Helps Supervised Neural Word Segmentation

By exploiting unlabeled data for further performance improvement for Chinese word segmentation, this work makes the first attempt at exploring adding unsupervised segmentation information into neural supervised segmenter. We survey various effective strategies, including extending the character embedding, augmenting the word score and applying multi-task learning, for leveraging unsupervised information derived from abundant unlabeled data. Experiments on standard data sets show that the explored strategies indeed improve the recall rate of out-of-vocabulary words and thus boost the segmentation accuracy. Moreover, the model enhanced by the proposed methods outperforms state-of-theart models in closed test and shows promising improvement trend when adopting three different strategies with the help of a large unlabeled data set. Our thorough empirical study eventually verifies the proposed approach outperforms the widelyused pre-training approach in terms of effectively making use of freely abundant unlabeled data.

A New Ensemble Learning Framework for 3D Biomedical Image Segmentation

3D image segmentation plays an important role in biomedical image analysis. Many 2D and 3D deep learning models have achieved state-of-the-art segmentation performance on 3D biomedical image datasets. Yet, 2D and 3D models have their own strengths and weaknesses, and by unifying them together, one may be able to achieve more accurate results. In this paper, we propose a new ensemble learning framework for 3D biomedical image segmentation that combines the merits of 2D and 3D models. First, we develop a fully convolutional network based meta-learner to learn how to improve the results from 2D and 3D models (base-learners). Then, to minimize over-fitting for our sophisticated meta-learner, we devise a new training method that uses the results of the baselearners as multiple versions of “ground truths”. Furthermore, since our new meta-learner training scheme does not depend on manual annotation, it can utilize abundant unlabeled 3D image data to further improve the model. Extensive experiments on two public datasets (the HVSMR 2016 Challenge dataset and the mouse piriform cortex dataset) show that our approach is effective under fully-supervised, semisupervised, and transductive settings, and attains superior performance over state-of-the-art image segmentation methods.

Joint Semi-Supervised Feature Selection and Classification through Bayesian Approach

With the increasing data dimensionality, feature selection has become a fundamental task to deal with high-dimensional data. Semi-supervised feature selection focuses on the problem of how to learn a relevant feature subset in the case of abundant unlabeled data with few labeled data. In recent years, many semi-supervised feature selection algorithms have been proposed. However, these algorithms are implemented by separating the processes of feature selection and classifier training, such that they cannot simultaneously select features and learn a classifier with the selected features. Moreover, they ignore the difference of reliability inside unlabeled samples and directly use them in the training stage, which might cause performance degradation. In this paper, we propose a joint semi-supervised feature selection and classification algorithm (JSFS) which adopts a Bayesian approach to automatically select the relevant features and simultaneously learn a classifier. Instead of using all unlabeled samples indiscriminately, JSFS associates each unlabeled sample with a self-adjusting weight to distinguish the difference between them, which can effectively eliminate the irrelevant unlabeled samples via introducing a left-truncated Gaussian prior. Experiments on various datasets demonstrate the effectiveness and superiority of JSFS.

Person Reidentification via Multi-Feature Fusion With Adaptive Graph Learning.

The goal of person reidentification (Re-ID) is to identify a given pedestrian from a network of nonoverlapping surveillance cameras. Most existing works follow the supervised learning paradigm which requires pairwise labeled training data for each pair of cameras. However, this limits their scalability to real-world applications where abundant unlabeled data are available. To address this issue, we propose a multi-feature fusion with adaptive graph learning model for unsupervised Re-ID. Our model aims to negotiate comprehensive assessment on the consistent graph structure of pedestrians with the help of special information of feature descriptors. Specifically, we incorporate multi-feature dictionary learning and adaptive multi-feature graph learning into a unified learning model such that the learned dictionaries are discriminative and the subsequent graph structure learning is accurate. An alternating optimization algorithm with proved convergence is developed to solve the final optimization objective. Extensive experiments on four benchmark data sets demonstrate the superiority and effectiveness of the proposed method.

Weakly Supervised Segmentation of SAR Imagery Using Superpixel and Hierarchically Adversarial CRF

Synthetic aperture radar (SAR) image segmentation aims at generating homogeneous regions from a pixel-based image and is the basis of image interpretation. However, most of the existing segmentation methods usually neglect the appearance and spatial consistency during feature extraction and also require a large number of training data. In addition, pixel-based processing cannot meet the real time requirement. We hereby present a weakly supervised algorithm to perform the task of segmentation for high-resolution SAR images. For effective segmentation, the input image is first over-segmented into a set of primitive superpixels. This algorithm combines hierarchical conditional generative adversarial nets (CGAN) and conditional random fields (CRF). The CGAN-based networks can leverage abundant unlabeled data learning parameters, reducing their reliance on the labeled samples. In order to preserve neighborhood consistency in the feature extraction stage, the hierarchical CGAN is composed of two sub-networks, which are employed to extract the information of the central superpixels and the corresponding background superpixels, respectively. Afterwards, CRF is utilized to perform label optimization using the concatenated features. Quantified experiments on an airborne SAR image dataset prove that the proposed method can effectively learn feature representations and achieve competitive accuracy to the state-of-the-art segmentation approaches. More specifically, our algorithm has a higher Cohen’s kappa coefficient and overall accuracy. Its computation time is less than the current mainstream pixel-level semantic segmentation networks.

Recursive Maximum Margin Active Learning

Active learning is an important technique to alleviate the problem when there is abundant unlabeled data but scarce labeled data. It aims to choose the most valuable samples to label in order to build powerful predictive models with minimal supervision. However, under the setup of active learning, when the data are characterized by high-dimensional features, it will be difficult to get a reliable estimate on the model parameters, as the labeled data are limited. Most existing works tackle this problem by learning a low-dimensional representation of data before active learning, but it cannot guarantee to obtain promising results, as traditional feature extraction techniques and active learning algorithms are designed independently. In this paper, we propose an efficient hybrid active learning algorithm, called Recursive Maximum Margin Active Learning. We optimize the active learning and semi-supervised feature extraction under a unified framework to tackle the high-dimensional features' problems and select the most representative samples in the low-dimensional space. By introducing the semi-supervised maximum margin criterion into active learning, we can conduct sample selection and feature extraction recursively at each iteration of active learning to learn more accurate models. The extensive experimental results show that the proposed method outperforms several state-of-the-art active learning methods on publicly available datasets.

A Semi-Supervised CNN With Fuzzy Rough C-Mean for Image Classification

While deep learning (DL) has proven to be a powerful paradigm for the classification of the large-scale big image data set. Deep learning network such as CNN requires a large number of labeled samples for training the network. However, often times the labeled data are difficult, expensive, and time-consuming. In this paper, we propose a semi-supervised approach to fused fuzzy-rough C-mean clustering with convolutional neural networks (CNNs) to knowledge learning from simultaneously intra-model and inter-model relationships forming final data representation to be classified, which can be achieved by better performance. Conception behind this is to reduce the uncertainty in terms of vagueness and indiscernibility by using fuzzy-rough C-mean clustering and specifically removing noise samples by using CNN from raw data. The framework of our proposed semi-supervised approach uses all the training data with abundant unlabeled data with few labeled data to train FRCNN model. To show the effectiveness of our model, we used four benchmark large-scale image datasets and also compared it with the state-of-the-art supervised, unsupervised, and semi-supervised learning methods for image classification.

Domain Adversarial for Acoustic Emotion Recognition

The performance of speech emotion recognition is affected by the differences in data distributions between train (source domain) and test (target domain) sets used to build and evaluate the models. This is a common problem, as multiple studies have shown that the performance of emotional classifiers drops when they are exposed to data that do not match the distribution used to build the emotion classifiers. The difference in data distributions becomes very clear when the training and testing data come from different domains, causing a large performance gap between development and testing performance. Due to the high cost of annotating new data and the abundance of unlabeled data, it is crucial to extract as much useful information as possible from the available unlabeled data. This study looks into the use of adversarial multitask training to extract a common representation between train and test domains. The primary task is to predict emotional-attribute-based descriptors for arousal, valence, or dominance. The secondary task is to learn a common representation, where the train and test domains cannot be distinguished. By using a gradient reversal layer, the gradients coming from the domain classifier are used to bring the source and target domain representations closer. We show that exploiting unlabeled data consistently leads to better emotion recognition performance across all emotional dimensions. We visualize the effect of adversarial training on the feature representation across the proposed deep learning architecture. The analysis shows that the data representations for the train and test domains converge as the data are passed to deeper layers of the network. We also evaluate the difference in performance when we use a shallow neural network versus a deep neural network and the effect of the number of shared layers used by the task and domain classifiers.

Semi-supervised cross-modal learning for cross modal retrieval and image annotation

Multimedia data are usually associated with multiple modalities represented by heterogeneous features. Recently, many information retrieval tasks are not only restricted to the case of a single modal and the contend-based cross modal retrieval has become one of the popular research fields. The premise of cross modal retrieval is discovering the relationships between different modalities efficiently. Though some approaches have been proposed to address this challenging problem, they either ignores the precious labels, or heavily depends on the completely labeled training data. In addition, for features with relatively high dimensionality, it is of great importance to select the most informative ones. In this paper, we propose a semi-supervised algorithm for cross modal learning. Our algorithm can make full use of both a small number of labeled and an abundant unlabeled data to establish connections between modalities via a shared semantic space discovering. On the other hand, our algorithm automatically filter out the noisy and redundant features to further improve our model. Finally, we give an efficient solution to the objective function. The experiments on two publicly available datasets demonstrate that the proposed method is competitive with or even superior to the state-of-art counterparts.

Semi-Supervised Deep Learning Classification for Hyperspectral Image Based on Dual-Strategy Sample Selection

This paper studies the classification problem of hyperspectral image (HSI). Inspired by the great success of deep neural networks in Artificial Intelligence (AI), researchers have proposed different deep learning based algorithms to improve the performance of hyperspectral classification. However, deep learning based algorithms always require a large-scale annotated dataset to provide sufficient training. To address this problem, we propose a semi-supervised deep learning framework based on the residual networks (ResNets), which use very limited labeled data supplemented by abundant unlabeled data. The core of our framework is a novel dual-strategy sample selection co-training algorithm, which can successfully guide ResNets to learn from the unlabeled data by making full use of the complementary cues of the spectral and spatial features in HSI classification. Experiments on the benchmark HSI dataset and real HSI dataset demonstrate that, with a small number of training data, our approach achieves competitive performance with maximum improvement of 41% (compare with traditional convolutional neural network (CNN) with 5 initial training samples per class on Indian Pines dataset) for HSI classification as compared with the results from those state-of-the-art supervised and semi-supervised methods.

Unsupervised Feature Learning with Single Layer ICANet for Face Recognition

Compared to supervised learning, unsupervised learning allows systems to learn consistent patterns from cheap and abundant unlabeled data, without the need for manual annotation. In this paper, we present a novel unsupervised feature learning method with single layer (SL) network based independent component analysis (ICA) filters called SL-ICANet, with the goal of achieving compact and robust facial feature representation. Our contributions are twofold: (i) We developed a single-layer convolutional network for unsupervised learning wherein the trainable kernels are replaced by ICA filters. (ii) We extended our SL-ICANet to use multi-scale information for better feature learning. Extensive experiments on two popular face recognition benchmarks, namely, labeled faces in the wild and facial recognition technology show that the proposed method might serve as a simple but highly competitive baseline for face recognition.

DEMIAL: an active learning framework for multiple instance image classification using dictionary ensembles

In many machine-learning applications, each data point can be represented as a set of instances that create multiple instance learning (MIL) problems. Due to the structure of images, different regions can be interpreted as instances. Thus, multiple instances can be obtained for each image, which makes image categorization a MIL problem. With abundant unlabeled image data, this MIL problem can be solved using active learning algorithms. Active learning is a framework that utilizes unlabeled data in which labeling samples is a labor-intensive and expensive task. Although many effective MIL active learning methods have been developed, most of the existing algorithms do not take into account classifier and feature representation. In this work, we develop DEMIAL (dictionary ensembles multiple instance active learning), a multiple instance active learning method that utilizes sparse feature representation and classifier ensemble techniques. In the proposed active learning framework, we employ dictionary learning and compare uncertainty- and entropy-based instance selection techniques. Experimental results show that classifier ensembles benefit from active learning and the DEMIAL algorithm outperforms the kernel-based multiple instance active learning framework.

Semi-Supervised Deep Learning Using Pseudo Labels for Hyperspectral Image Classification.

Deep learning has gained popularity in a variety of computer vision tasks. Recently, it has also been successfully applied for hyperspectral image classification tasks. Training deep neural networks, such as a convolutional neural network for classification requires a large number of labeled samples. However, in remote sensing applications, we usually only have a small amount of labeled data for training because they are expensive to collect, although we still have abundant unlabeled data. In this paper, we propose semi-supervised deep learning for hyperspectral image classification-our approach uses limited labeled data and abundant unlabeled data to train a deep neural network. More specifically, we use deep convolutional recurrent neural networks (CRNN) for hyperspectral image classification by treating each hyperspectral pixel as a spectral sequence. In the proposed semi-supervised learning framework, the abundant unlabeled data are utilized with their pseudo labels (cluster labels). We propose to use all the training data together with their pseudo labels to pre-train a deep CRNN, and then fine-tune using the limited available labeled data. Further, to utilize spatial information in the hyperspectral images, we propose a constrained Dirichlet process mixture model (C-DPMM), a non-parametric Bayesian clustering algorithm, for semi-supervised clustering which includes pairwise must-link and cannot-link constraints-this produces high-quality pseudo-labels, resulting in improved initialization of the deep neural network. We also derived a variational inference model for the C-DPMM for efficient inference. Experimental results with real hyperspectral image data sets demonstrate that the proposed semi-supervised method outperforms state-of-the-art supervised and semi-supervised learning methods for hyperspectral classification.