Unlabeled Instances Research Articles

A weighted ensemble-based active learning model to label microarray data.

Classification of cancerous genes from microarray data is an important research area in bioinformatics. Large amount of microarray data are available, but it is very costly to label them. This paper proposes an active learning model, a semi-supervised classification approach, to label the microarray data using which predictions can be made even with lesser amount of labeled data. Initially, a pool of unlabeled instances is given from which some instances are randomly chosen for labeling. Successive selection of instances to be labeled from unlabeled pool is determined by selection algorithms. The proposed method is devised following an ensemble approach to combine the decisions of three classifiers in order to arrive at a consensus which provides a more accurate prediction of the class label to ensure that each individual classifier learns in an uncorrelated manner. Our method combines the heuristic techniques used by an active learning algorithm to choose training samples with the multiple learning paradigm attained by an ensemble to optimize the search space by choosing efficiently from an already sparse learning pool. On evaluating the proposed method on 10 microarray datasets, we achieve performance which is comparable with state-of-the-art methods. The code and datasets are given at https://github.com/anuran-Chakraborty/Active-learning. Flowchart of the proposed ensemble-based active learning framework.

Particle Swarm Optimization Based Swarm Intelligence for Active Learning Improvement: Application on Medical Data Classification

Semi-supervised learning targets the common situation where labeled data are scarce but unlabeled data are abundant. It uses unlabeled data to help supervised learning tasks. In practice, it may make sense to utilize active learning in conjunction with semi-supervised learning. That is, we might allow the learning algorithm to pick a set of unlabeled instances to be labeled by a domain expert, which will then be used as the labeled data set. However, existing approaches are computationally expensive and require searching through an entire unlabeled dataset, which may contain redundant instances that provide no instructive information to the classifier and can decrease the performance. To address this optimization problem, a hybrid system that combines active learning (AL) and particle swarm optimization (PSO) algorithms is proposed to reduce the cost of labeling while building a more efficient classifier. The novelty of this work resides in the integration of a bio-inspired optimization algorithm in the machine learning strategy. Furthermore, a novel uncertainty measure was integrated into the particle swarm optimization algorithm as an objective function to select from massive amounts of medical instances those that are deemed most informative. To evaluate the effectiveness of the proposed approach, eighteen (18) benchmark datasets were used and compared against three best-known classifiers with different learning paradigms: AL–NB an active learning algorithm using Naive Base classifier and Margin Sampling strategy, SVM (Support Vector Machine), ELM (Extreme Learning Machine) with supervised learning, and TSVM (Transductive Support Vector Machine) with the semi-supervised learning. Experiments showed that the proposed approach is effective in reducing the efforts required by experts for medical data annotation to produce an accurate classifier. The active learning approach has been utilized to optimize the expensive task of labeling. Based on a novel uncertainty measure, the nature-inspired algorithm PSO attempts to select from massive amounts of unlabeled medical instances those considered informative, at the same time improving the classifier performance. The experiments carried out confirm that the proposed strategy significantly enhances the performance of the AL algorithm compared with the commonly used uncertainty strategies. It achieves a performance similar to that of fully supervised and semi-supervised algorithms while requiring much less labeling. As a future extension of this work, it would be interesting to integrate other evolutionary optimization algorithms and compare them with our approach. In addition, it is beneficial to test the impact of using other variants of PSO algorithm in our approach. Also, it is aimed to test more classification algorithms in the experimentation process.

Joint Multi-Dimensional Model for Global and Time-Series Annotations

Crowdsourcing is a popular approach to collect annotations for unlabeled data instances. It involves collecting a large number of annotations from several, often naive untrained annotators for each data instance which are then combined to estimate the ground truth. Further, annotations for constructs such as affect are often multi-dimensional with annotators rating multiple dimensions, such as valence and arousal, for each instance. Most annotation fusion schemes however ignore this aspect and model each dimension separately. In this article we address this by proposing a generative model for multi-dimensional annotation fusion, which models the dimensions jointly leading to more accurate ground truth estimates. The model we propose is applicable to both global and time series annotation fusion problems and treats the ground truth as a latent variable distorted by the annotators. The model parameters are estimated using the Expectation-Maximization algorithm and we evaluate its performance using synthetic data and real emotion corpora as well as on an artificial task with human annotations.

Active Ensemble Deep Learning for Polarimetric Synthetic Aperture Radar Image Classification

Although deep learning has achieved great success in image classification tasks, its performance is subject to the quantity and quality of training samples. For classification of polarimetric synthetic aperture radar (PolSAR) images, it is nearly impossible to annotate the images from visual interpretation. Therefore, it is urgent for remote sensing scientists to develop new techniques for PolSAR image classification under the condition of very few training samples. In this letter, we take the advantage of active learning and propose active ensemble deep learning (AEDL) for PolSAR image classification. We first show that only 35\% of the predicted labels of a deep learning model's snapshots near its convergence were exactly the same. The disagreement between snapshots is non-negligible. From the perspective of multiview learning, the snapshots together serve as a good committee to evaluate the importance of unlabeled instances. Using the snapshots committee to give out the informativeness of unlabeled data, the proposed AEDL achieved better performance on two real PolSAR images compared with standard active learning strategies. It achieved the same classification accuracy with only 86% and 55% of the training samples compared with breaking ties active learning and random selection for the Flevoland dataset.

Structural damage identification via physics-guided machine learning: a methodology integrating pattern recognition with finite element model updating

Structural health monitoring methods are broadly classified into two categories: data-driven methods via statistical pattern recognition and physics-based methods through finite elementmodel updating. Data-driven structural health monitoring faces the challenge of data insufficiency that renders the learned model limited in identifying damage scenarios that are not contained in the training data. Model-based methods are susceptible to modeling error due to model idealizations and simplifications that make the finite element model updating results deviate from the truth. This study attempts to combine the merits of data-driven and physics-based structural health monitoring methods via physics-guided machine learning, expecting that the damage identification performance can be improved. Physics-guided machine learning uses observed feature data with correct labels as well as the physical model output of unlabeled instances. In this study, physics-guided machine learning is realized with a physics-guided neural network. The original modal-property based features are extended with the damage identification result of finite element model updating. A physics-based loss function is designed to evaluate the discrepancy between the neural network model output and that of finite element model updating. With the guidance from the scientific knowledge contained in finite element model updating, the learned neural network model has the potential to improve the generality and scientific consistency of the damage detection results. The proposed methodology is validated by a numerical case study on a steel pedestrian bridge model and an experimental study on a three-story building model.

Root Cause Analysis for Self-organizing Cellular Network: an Active Learning Approach

To ease the configuration and maintenance of complex cellular networks, the self-organizing network (SON) is introduced. SON contains three major sub-functional groups: self-configuration, self-optimization, and self-healing. Among these, fault diagnosis in self-healing is crucial, and it is usually considered as a classification problem which is commonly addressed by supervised machine learning methods. However, the barrier to these methods is the difficulty in obtaining sufficient network fault data with label (fault cause). To achieve an effective classifier and dramatically reduce the number of labeled instances needed, we propose an active learning based fault diagnosis scheme, which can select unlabeled instances for labeling actively. According to the selection criteria, there are several query strategies. In this paper, we apply uncertainty sampling as the query strategy due to its low computational cost and high efficiency. Besides, we implement random sampling as a contrast which is a nonactive learning method. To verify the effectiveness of the proposed scheme, we construct a long term evolution (LTE) system level simulator by Network Simulator 3. Then several fault scenarios are simulated, and the records of key performance indicators with fault causes are collected. Extensive experiments demonstrate that the proposed scheme is effective in reducing the number of labeled instances needed, and it is also valid in the class-imbalanced data. Specifically, to achieve a classifier with an accuracy of 99%, the active learning based method only needs 74 labeled instances but the nonactive learning method needs 1354 ones.

A new transfer learning-based method for label proportions problem

Learning with label proportions (LLP), which only provides the unlabeled instances in the bag and the bag’s label proportion, has been widely studied recently. However, most of the existing LLP methods do not consider the knowledge transfer from the source task to the target task. In addition, in the process of the collecting data, the data may be corrupted by noise, and this always leads to the uncertain data in its representation. This paper proposes a new transfer learning-based approach for the problem of learning with label proportions, which is called TL-LLP in brief, to transfer knowledge from the source task to the target task where both the source and target tasks contain uncertain data. We first formulate the objective model to deal with transfer learning and uncertain data for the label proportions problem at the same time. We then propose an iterative framework to solve the proposed objective model and obtain the accurate classifier for the target task. Extensive experiments have shown that the proposed TL-LLP method can obtain better performance and is less sensitive to noise compared with the existing LLP methods.

Masking domain-specific information for cross-domain deception detection

The facilities provided by social media and computer-mediated communication make easy the dissemination of deceptive behavior, after which different entities or people could be affected. The deception detection by supervised learning has been widely studied; however, the scenario in which there is one domain of interest and the labeled data is in another domain has received poor attention. This paper presents, to our knowledge, the first domain adaptation approach for cross-domain deception detection in texts. Our proposal consists in modifying original texts from the source and target domains in a form in which common content and style information is maintained, but domain-specific information is masked. In order to adequately select domain-specific terms to be masked, the proposed method uses unlabeled instances from both domains. Our experiments demonstrate that the masking technique is a good idea for detecting deception in cross-domain scenarios; and the performance could be further improved if unlabeled information from the target domain is considered.

End-to-end relation extraction based on bootstrapped multi-level distant supervision

Distant supervised relation extraction has been widely used to identify new relation facts from free text, since the existence of knowledge base helps these models to build a large dataset with few human intervention and low costs of manpower and time. However, the existing Distant Supervised models are all based on the single-node classifier so that they suffer from the serious false categorization problem especially for the existence of thousands of relations. In this paper, we novelly propose an end-to-end model for relation extraction based on distant supervision. Our model divides the original categorization task into a number of sub-tasks which focus on the construction of tree-like categorization structure in multiple levels. With the tree-like structure, an unlabelled relation instance can be categorized step by step along a path from the root node to a leaf node. An additional benefit of this structure is that it can be used to select negative samples from training data for each child node. In addition, to the best of our knowledge, no effort has been made to update the categorization model with new identified relation facts, which hinders the improvement of the extraction precision and recall. Although bootstrapping methods can contribute to improving the situation, they need additional calculation to evaluate the quality of extracted patterns or tuples when selecting new instances for next iterations. In this paper, we propose to do bootstrapped distant supervision to update the distant supervision model with new learned relation facts iteratively, and we can use scores directly gotten in the model to evaluate the quality of instances instead of additional calculation. As a result, we can further improve the extraction precision and recall. To save the time costs and manpower, we also propose an adaptive method by mapping function to choose the suitable thresholds for each iteration without manual choice rather than using the fixed thresholds. Experimental results conducted on three real datasets prove that our approach outperforms state-of-the-art approaches by reaching 12+% better extraction quality.

Heterogeneous Cross Project Defect Prediction in Software

Software defect prediction is one of the software engineering's most active research fields. Most of the existing work focuses on Homogeneous Cross Project Defect Prediction (CPDP), in which the model is trained by the use of a common metric set extracted from the source and target project. Our article emphasizes the heterogeneous CPDP modeling (HCPDP) that develops a defect prediction model based on a metric choice and metric matching strategy and also shows a comparable value distribution for a specified couple of datasets. It assesses experimentally and theoretically HCPDP modeling whose three primary elements include feature ranking and feature selection, metric matching an binary classification of unlabeled target instances. Results indicate that feature selection techniques have a very tiny effect on the performance of defect prediction, and the Gradient Boosting classification system provides the highest findings when used in combination with the Pearson Correlation technique compared to other classifiers used.

Divide-and-conquer ensemble self-training method based on probability difference

Self-training method can train an effective classifier by exploiting labeled instances and unlabeled instances. In the process of self-training method, the high confidence instances are usually selected iteratively and added to the training set for learning. Unfortunately, the structure information of high confidence instances is so similar that it leads to local over-fitting during the iterations. In order to avoid the over-fitting phenomenon, and improve the classification effect of self-training methods, a novel divide-and-conquer ensemble self-training framework based on probability difference is proposed. Firstly, the probability difference of instances is calculated by the category probability of each classifier, the low-fuzzy and high-fuzzy instances of each classifier are divided through the probability difference. Then, a divide-and-conquer strategy is adopted. That is, the low-fuzzy instances determined by all the classifiers are directly labeled and high-fuzzy instances are manually labeled. Finally, the labeled instances are added to the training set for iteration self-training. This method expands the training set by selecting low-fuzzy instances with accurate structure information and high-fuzzy instances with more comprehensive structure information, and it improves the generalization performance of the method effectively. The method is more suitable for noise data sets and it can obtain structure information even in a few labeled instances. The effectiveness of the proposed method is verified by comparative experiments on the University of California Irvine (UCI).

Deep transductive network for generalized zero shot learning

Zero Shot Learning (ZSL) aims to learn projective functions on labeled seen data and transfer the learned functions to unseen classes by discovering their relationship with semantic embeddings. However, the mapping process often suffers from the domain shift problem caused by only using the labeled seen data. In this paper, we propose a novel explainable Deep Transductive Network (DTN) for the task of Generalized ZSL (GZSL) by training on both labeled seen data and unlabeled unseen data, with subsequent testing on both seen classes and unseen classes. The proposed network exploits a KL Divergence constraint to iteratively refine the probability of classifying unlabeled instances by learning from their high confidence assignments with the assistance of an auxiliary target distribution. Besides, to avoid the meaningless ascription assumption of unseen data on GZSL, we also propose an experimental paradigm by splitting the unseen data into two equivalent parts for training and testing respectively. Extensive experiments and detailed analysis demonstrate that our DTN can efficiently handle the problems and achieve the state-of-the-art performance on four popular datasets.

Improving Neural Relation Extraction with Positive and Unlabeled Learning

We present a novel approach to improve the performance of distant supervision relation extraction with Positive and Unlabeled (PU) Learning. This approach first applies reinforcement learning to decide whether a sentence is positive to a given relation, and then positive and unlabeled bags are constructed. In contrast to most previous studies, which mainly use selected positive instances only, we make full use of unlabeled instances and propose two new representations for positive and unlabeled bags. These two representations are then combined in an appropriate way to make bag-level prediction. Experimental results on a widely used real-world dataset demonstrate that this new approach indeed achieves significant and consistent improvements as compared to several competitive baselines.

Semi-Supervised Streaming Learning with Emerging New Labels

In many real-world applications, the modeling environment is usually dynamic and evolutionary, especially in a data stream where emerging new class often happens. Great efforts have been devoted to learning with novel concepts recently, which are typically in a supervised setting with completely supervised initialization. However, the data collected in the stream are often in a semi-supervised manner actually, which means only a few of them are labeled while the great majority miss ground-truth labels. Besides, new classes hidden in unlabeled instances bring more challenges for the learning task. In this paper, we tackle these issues by a new approach called SEEN which consists of three major components: an effective novel class detector based on clustering random trees, a robust classifier for predictions on the known classes, and an efficient updating process that ensures the whole framework adapts to the changing environment automatically. The classifier produces known labels via label propagation that utilizes all labeled and part unlabeled data in the past which naturally describe the entire stream seen so far. Empirical studies on several datasets validate that the algorithm can accurately classify points on a dynamic stream with a small number of labeled examples and emerging new classes.

Asking the Right Questions to the Right Users: Active Learning with Imperfect Oracles

Active learning algorithms automatically identify the salient and exemplar samples from large amounts of unlabeled data and tremendously reduce human annotation effort in inducing a machine learning model. In a traditional active learning setup, the labeling oracles are assumed to be infallible, that is, they always provide correct answers (in terms of class labels) to the queried unlabeled instances. However, in real-world applications, oracles are often imperfect and provide incorrect label annotations. Oracles also have diverse expertise and while they may be noisy, certain oracles may provide accurate annotations to certain specific instances. In this paper, we propose a novel framework to address the challenging problem of active learning in the presence of multiple imperfect oracles. We pose the optimal sample and oracle selection as a constrained optimization problem and derive a linear programming relaxation to select a batch of (sample-oracle) pairs, which can potentially augment maximal information to the underlying classification model. Our extensive empirical studies on 9 challenging datasets (from a variety of application domains) corroborate the usefulness of our framework over competing baselines.

A hybrid safe semi-supervised learning method

Within the past few years, Safe Semi-Supervised Learning (S3L) has become a hot topic in the machine learning field and many related S3L methods have been proposed to safely exploit the unlabeled information. However, these methods only considered the risk from a single level, such as the instance or model level. They can not reduce the adverse effects of both the risky unlabeled instances and inappropriate learning models. Therefore, it is important to investigate a novel effective S3L method. In this paper, we present a hybrid S3L method which can inherit the merits of both the instance-level and model-level approaches. In our algorithm, multiple Graph-based SSL (GSSL) classifiers are firstly trained and used to predict the unlabeled instances. The risk degrees of the unlabeled instances and the qualities of the constructed graphs are then estimated through the predictions of multiple GSSL classifiers. Finally, we build two regularization terms to constrain the predictions of the unlabeled instances and adaptively select the graphs with high qualities. These regularization terms aim at reducing the negative effect of both the risky unlabeled instances and inappropriate learning models with low-quality graphs. Experimental results on different real-world datasets verify the effectiveness of our algorithm by comparisons to the state-of-the-art Supervised Learning (SL), SSL and S3L methods. In conclusion, our algorithm can not only enrich the research of S3L, but enlarge the practical scope of SSL in the expert and intelligent systems to a certain extent.

Regularization on a rapidly varying manifold

The presence of inflection points on data manifold or rapid variation makes it difficult for the second order derivative based graph Laplacian and Hessian regularization techniques to accurately approximate the marginal distribution parameters. Moreover, in general, function over-fitting on seen unlabeled instances due to lack of extrapolation power which makes graph Laplacian regularization based solution biased towards constant. Hessian solves this problem by opting a generic function based on the function’s divergence in more than one direction. However, due to the presence of inflection points in the dense region, the function remains unpenalized by Hessian manifold regularization. We propose a Jerk based manifold regularization (JR) for dense, oscillating and manifolds with inflection points. JR approximates the rate of change of curvature from the underlying manifold which appropriately identifies the unpenalized geodesic deviating functions and accurately penalizes them. It also helps in identifying the optimal function in the presence of inflection points. Extensive experiments on synthetic and real-world datasets show that the proposed JR technique approximates accurate and generic input space geometrical constraints to outperform existing state-of-the-art manifold regularization techniques by a significant margin.

Conformal prediction based active learning by linear regression optimization

Conformal prediction uses the degree of strangeness (nonconformity) of data instances to determine the confidence values of new predictions. We propose a conformal prediction based active learning algorithm, referred to as CPAL-LR, to improve the performance of pattern classification algorithms. CPAL-LR uses a novel query function that determines the relevance of unlabeled instances through the solution of a constrained linear regression model, incorporating uncertainty, diversity, and representativeness in the optimization problem. Furthermore, we present a nonconformity measure that produces reliable confidence values. CPAL-LR is implemented in conjunction with support vector machines, sparse coding algorithms, and convolutional networks. Experiments conducted on face and object recognition databases demonstrate that CPAL-LR improves the classification performance of a variety classifiers, outperforming previously proposed active learning techniques, while producing reliable confidence values.

An Incremental Kernel Extreme Learning Machine for Multi-Label Learning With Emerging New Labels

Multi-label learning with emerging new labels is a practical problem that occurs in data streams and has become an important new research issue in the area of machine learning. However, existing models for dealing with this problem require high learning computational times, and there still exists a lack of research. Based on these issues, this paper presents an incremental kernel extreme learning machine for multi-label learning with emerging new labels, consisting of two parts: a novelty detector; and a multi-label classifier. The detector with free-user-setting threshold parameters was developed to identify instances with new labels. A new incremental multi-label classifier and its improved version were developed to predict a label set for each instance, which can add output units incrementally and update themselves in unlabeled instances. Comprehensive evaluations of the proposed method were carried out on the problems of multi-label classification with emerging new labels compared to comparative algorithms, which revealed the promising performance of the proposed method.

Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach.

Unsupervised domain adaptation (uDA) models focus on pairwise adaptation settings where there is a single, labeled, source and a single target domain. However, in many real-world settings one seeks to adapt to multiple, but somewhat similar, target domains. Applying pairwise adaptation approaches to this setting may be suboptimal, as they fail to leverage shared information among multiple domains. In this work, we propose an information theoretic approach for domain adaptation in the novel context of multiple target domains with unlabeled instances and one source domain with labeled instances. Our model aims to find a shared latent space common to all domains, while simultaneously accounting for the remaining private, domain-specific factors. Disentanglement of shared and private information is accomplished using a unified information-theoretic approach, which also serves to establish a stronger link between the latent representations and the observed data. The resulting model, accompanied by an efficient optimization algorithm, allows simultaneous adaptation from a single source to multiple target domains. We test our approach on three challenging publicly-available datasets, showing that it outperforms several popular domain adaptation methods.