Use Of Unlabeled Data Research Articles

CUPID: consistent unlabeled probability of identical distribution for image classification

We consider the general problem of learning from both labeled and unlabeled data, which is often called semi-supervised learning (SSL) or transductive inference. A principled approach to SSL is to design a classification function that is sufficiently smooth with respect to the underlying structure collectively revealed by known labeled and unlabeled data. Combining transductive learning and inductive learning together, we present a simple and scalable algorithm to obtain such a smooth function, namely, Consistent Unlabeled Probability of Identical Distribution (CUPID). The labels of unlabeled data are taken as the probability, consistent to their identical distribution based on geometric structure of the unlabeled. The proposed algorithm yields encouraging experimental results on a number of image classification problems and demonstrates effective use of unlabeled data.

A Framework for pre-training hidden-unit conditional random fields and its extension to long short term memory networks

In this paper, we introduce a simple unsupervised framework for pre-training hidden-unit conditional random fields (HUCRFs), i.e., learning initial parameter estimates for HUCRFs prior to supervised training.Our framework exploits the model structure of HUCRFs to make effective use of unlabeled data from the same domain or labeled data from a different domain. The key idea is to use the separation of HUCRF parameters between observations and labels: this allows us to pre-train observation parameters independently of label parameters. Pre-training is achieved by creating pseudo-labels from such resources. In the case of unlabeled data, we cluster observations and use the resulting clusters as pseudo-labels. Observation parameters can be trained on these resources and then transferred to initialize the supervised training process on the target labeled data. Experiments on various sequence labeling tasks demonstrate that the proposed pre-training method consistently yields significant improvement in performance. The core idea could be extended to other learning techniques including deep learning. We applied the proposed technique to recurrent neural networks (RNN) with long short term memory (LSTM) architecture and obtained similar gains.

A novel cross-modal hashing algorithm based on multimodal deep learning

With the growing popularity of multimodal data on the Web, cross-modal retrieval on large-scale multimedia databases has become an important research topic. Cross-modal retrieval methods based on hashing assume that there is a latent space shared by multimodal features. To model the relationship among heterogeneous data, most existing methods embed the data into a joint abstraction space by linear projections. However, these approaches are sensitive to noise in the data and are unable to make use of unlabeled data and multimodal data with missing values in real-world applications. To address these challenges, we proposed a novel multimodal deep-learning-based hash (MDLH) algorithm. In particular, MDLH uses a deep neural network to encode heterogeneous features into a compact common representation and learns the hash functions based on the common representation. The parameters of the whole model are fine-tuned in a supervised training stage. Experiments on two standard datasets show that the method achieves more effective results than other methods in cross-modal retrieval.

An empirical study of self-training and data balancing techniques for splice site prediction

Thanks to Next Generation Sequencing technologies, unlabelled data is now generated easily, while the annotation process remains expensive. Semi-supervised learning represents a cost-effective alternative to supervised learning, as it can improve supervised classifiers by making use of unlabelled data. However, semi-supervised learning has not been studied much for problems with highly skewed class distributions, which are prevalent in bioinformatics. To address this limitation, we carry out a study of a semi-supervised learning algorithm, specifically self-training based on Naive Bayes, with focus on data-level approaches for handling imbalanced class distributions. Our study is conducted on the problem of predicting splice sites and it is based on datasets for which the ratio of positive to negative examples is 1-to-99. Our results show that under certain conditions semi-supervised learning algorithms are a better choice than purely supervised classification algorithms.

Semi-supervised learning for ordinal Kernel Discriminant Analysis

Ordinal classification considers those classification problems where the labels of the variable to predict follow a given order. Naturally, labelled data is scarce or difficult to obtain in this type of problems because, in many cases, ordinal labels are given by a user or expert (e.g. in recommendation systems). Firstly, this paper develops a new strategy for ordinal classification where both labelled and unlabelled data are used in the model construction step (a scheme which is referred to as semi-supervised learning). More specifically, the ordinal version of kernel discriminant learning is extended for this setting considering the neighbourhood information of unlabelled data, which is proposed to be computed in the feature space induced by the kernel function. Secondly, a new method for semi-supervised kernel learning is devised in the context of ordinal classification, which is combined with our developed classification strategy to optimise the kernel parameters. The experiments conducted compare 6 different approaches for semi-supervised learning in the context of ordinal classification in a battery of 30 datasets, showing (1) the good synergy of the ordinal version of discriminant analysis and the use of unlabelled data and (2) the advantage of computing distances in the feature space induced by the kernel function.

Asymptotic comparison of semi-supervised and supervised linear discriminant functions for heteroscedastic normal populations

It has been reported that using unlabeled data together with labeled data to construct a discriminant function works successfully in practice. However, theoretical studies have implied that unlabeled data can sometimes adversely affect the performance of discriminant functions. Therefore, it is important to know what situations call for the use of unlabeled data. In this paper, asymptotic relative efficiency is presented as the measure for comparing analyses with and without unlabeled data under the heteroscedastic normality assumption. The linear discriminant function maximizing the area under the receiver operating characteristic curve is considered. Asymptotic relative efficiency is evaluated to investigate when and how unlabeled data contribute to improving discriminant performance under several conditions. The results show that asymptotic relative efficiency depends mainly on the heteroscedasticity of the covariance matrices and the stochastic structure of observing the labels of the cases.

Semi-supervised Collective Classification in Multi-attribute Network Data

Multi-attribute network refers to network data with multiple attribute views and relational view. Although semi-supervised collective classification has been investigated extensively, little attention is received for such kind of network data. In this paper, we aim to study and solve the semi-supervised learning problem for multi-attribute networks. There are two important challenges: (1) how to extract effective information from the rich multi-attribute and relational information; (2) how to make use of unlabeled data in the network. We propose a new generative model with network regularization, called MARL, which addresses the two challenges. In the approach, a generative model based on the probabilistic latent semantic analysis method is developed to leverage attribute information, and a network regularizer is incorporated to smooth label probability with relational information and unlabeled data. Comprehensive experiments on various data sets have been conducted to demonstrate the effectiveness of the proposed MARL, and the results reveal that our approach outperforms existing collective classification methods and multi-view classification methods in terms of accuracy.

Towards Safe Semi-Supervised Learning for Multivariate Performance Measures

Semi-supervised learning (SSL) is an important research problem in machine learning. While it is usually expected that the use of unlabeled data can improve performance, in many cases SSL is outperformed by supervised learning using only labeled data. To this end, the construction of a performance-safe SSL method has become a key issue of SSL study. To alleviate this problem, we propose in this paper the UMVP (safe semi-sUpervised learning for MultiVariate Performance measure) method, because of the need of various performance measures in practical tasks. The proposed method integrates multiple semi-supervised learners, and maximizes the worst-case performance gain to derive the final prediction. The overall problem is formulated as a maximin optimization. In oder to solve the resultant difficult maximin optimization, this paper shows that when the performance measure is the Top-k Precision, Fβ score or AUC, a minimax convex relaxation of the maximin optimization can be solved efficiently. Experimental results show that the proposed method can effectively improve the safeness of SSL under multiple multivariate performance measures.

Semi-supervised prediction of gene regulatory networks using machine learning algorithms.

Use of computational methods to predict gene regulatory networks (GRNs) from gene expression data is a challenging task. Many studies have been conducted using unsupervised methods to fulfill the task; however, such methods usually yield low prediction accuracies due to the lack of training data. In this article, we propose semi-supervised methods for GRN prediction by utilizing two machine learning algorithms, namely, support vector machines (SVM) and random forests (RF). The semi-supervised methods make use of unlabelled data for training. We investigated inductive and transductive learning approaches, both of which adopt an iterative procedure to obtain reliable negative training data from the unlabelled data. We then applied our semi-supervised methods to gene expression data of Escherichia coli and Saccharomyces cerevisiae, and evaluated the performance of our methods using the expression data. Our analysis indicated that the transductive learning approach outperformed the inductive learning approach for both organisms. However, there was no conclusive difference identified in the performance of SVM and RF. Experimental results also showed that the proposed semi-supervised methods performed better than existing supervised methods for both organisms.

Semi-supervised deep extreme learning machine for Wi-Fi based localization

Along with the proliferation of mobile devices and wireless signal coverage, indoor localization based on Wi-Fi gets great popularity. Fingerprint based method is the mainstream approach for Wi-Fi indoor localization, for it can achieve high localization performance as long as labeled data are sufficient. However, the number of labeled data is always limited due to the high cost of data acquisition. Nowadays, crowd sourcing becomes an effective approach to gather large number of data; meanwhile, most of them are unlabeled. Therefore, it is worth studying the use of unlabeled data to improve localization performance. To achieve this goal, a novel algorithm Semi-supervised Deep Extreme Learning Machine (SDELM) is proposed, which takes the advantages of semi-supervised learning, Deep Leaning (DL), and Extreme Learning Machine (ELM), so that the localization performance can be improved both in the feature extraction procedure and in the classifier. The experimental results in real indoor environments show that the proposed SDELM not only outperforms other compared methods but also reduces the calibration effort with the help of unlabeled data.

An empirical study of ensemble-based semi-supervised learning approaches for imbalanced splice site datasets.

BackgroundRecent biochemical advances have led to inexpensive, time-efficient production of massive volumes of raw genomic data. Traditional machine learning approaches to genome annotation typically rely on large amounts of labeled data. The process of labeling data can be expensive, as it requires domain knowledge and expert involvement. Semi-supervised learning approaches that can make use of unlabeled data, in addition to small amounts of labeled data, can help reduce the costs associated with labeling. In this context, we focus on the problem of predicting splice sites in a genome using semi-supervised learning approaches. This is a challenging problem, due to the highly imbalanced distribution of the data, i.e., small number of splice sites as compared to the number of non-splice sites. To address this challenge, we propose to use ensembles of semi-supervised classifiers, specifically self-training and co-training classifiers.ResultsOur experiments on five highly imbalanced splice site datasets, with positive to negative ratios of 1-to-99, showed that the ensemble-based semi-supervised approaches represent a good choice, even when the amount of labeled data consists of less than 1% of all training data. In particular, we found that ensembles of co-training and self-training classifiers that dynamically balance the set of labeled instances during the semi-supervised iterations show improvements over the corresponding supervised ensemble baselines.ConclusionsIn the presence of limited amounts of labeled data, ensemble-based semi-supervised approaches can successfully leverage the unlabeled data to enhance supervised ensembles learned from highly imbalanced data distributions. Given that such distributions are common for many biological sequence classification problems, our work can be seen as a stepping stone towards more sophisticated ensemble-based approaches to biological sequence annotation in a semi-supervised framework.

A Novel Speech Emotion Recognition Method via Incomplete Sparse Least Square Regression

In this letter, we propose a novel speech emotion recognition method based on least square regression (LSR) model, in which a novel incomplete sparse LSR (ISLSR) model is proposed and utilized to characterize the linear relationship between speech features and the corresponding emotion labels. In training the ISLSR model, both labeled and unlabeled speech data sets are utilized, where the use of unlabeled data set aims to enhance the compatibility of the model such that it is well suitable for the out-of-sample speech data. Another novelty of ISLSR lies in the capability of dealing with feature selection. To evaluate the performance of the proposed method, we conduct experiments on two emotional speech databases. The experimental results on both databases demonstrate that the proposed method achieves better recognition performance in compared with several state-of-the-art methods.

A top-down information theoretic word clustering algorithm for phrase recognition

Semi-supervised machine learning methods have the features of both, integrating labeled and unlabeled training data. In most structural problems, such as natural language processing and image processing, developing labeled data for a specific domain requires considerable amount of human resources. In this paper, we present a cluster-based method to fuse labeled training and unlabeled raw data. We design a top-down divisive clustering algorithm that ensures maximal information gain in the use of unlabeled data via clustering similar words. To implement this idea, we design a top-down iterative K-means clustering algorithm to merge word clusters. Differently, the derived term groups are then encoded as new features for the supervised learners in order to improve the coverage of lexical information. Without additional training data or external materials, this approach yields state-of-the-art performance on the shallow parsing and base-chunking benchmark datasets (94.50 and 93.12 in F(β) rates).

Lexicon expansion for latent variable grammars

This study investigates the use of unlabeled data, i.e., raw texts, to strengthen latent variable probabilistic context-free grammars, in particular lexical models. A graph-based lexicon expansion approach is proposed to achieve this goal. It aims to discover additional lexical knowledge from a large amount of unlabeled data to help the syntax parsing. The proposed approach is based on a transductive graph-based label propagation technique. The approach builds k-nearest-neighbor (k-NN) similarity graphs over the words of labeled and unlabeled data, for propagating lexical emission probabilities. The intuition is that different word under similar syntactic environment should have approximate lexical emission distributions. The derived words, together with lexical emission probabilities, are incorporated into the parsing. This approach is very effective in parsing out-of-vocabulary (OOV) words. Empirical results for English, Chinese, and Portuguese revealed its effectiveness.

Predict Subcellular Locations of Singleplex and Multiplex Proteins by Semi-Supervised Learning and Dimension-Reducing General Mode of Chou's PseAAC

Predicting protein subcellular location is one of major challenges in Bioinformatics area since such knowledge helps us understand protein functions and enables us to select the targeted proteins during drug discovery process. While many computational techniques have been proposed to improve predictive performance for protein subcellular location, they have several shortcomings. In this work, we propose a method to solve three main issues in such techniques; i) manipulation of multiplex proteins which may exist or move between multiple cellular compartments, ii) handling of high dimensionality in input and output spaces and iii) requirement of sufficient labeled data for model training. Towards these issues, this work presents a new computational method for predicting proteins which have either single or multiple locations. The proposed technique, namely iFLAST-CORE, incorporates the dimensionality reduction in the feature and label spaces with co-training paradigm for semi-supervised multi-label classification. For this purpose, the Singular Value Decomposition (SVD) is applied to transform the high-dimensional feature space and label space into the lower-dimensional spaces. After that, due to limitation of labeled data, the co-training regression makes use of unlabeled data by predicting the target values in the lower-dimensional spaces of unlabeled data. In the last step, the component of SVD is used to project labels in the lower-dimensional space back to those in the original space and an adaptive threshold is used to map a numeric value to a binary value for label determination. A set of experiments on viral proteins and gram-negative bacterial proteins evidence that our proposed method improve the classification performance in terms of various evaluation metrics such as Aiming (or Precision), Coverage (or Recall) and macro F-measure, compared to the traditional method that uses only labeled data.

Multilabel relationship learning

Multilabel learning problems are commonly found in many applications. A characteristic shared by many multilabel learning problems is that some labels have significant correlations between them. In this article, we propose a novel multilabel learning method, called MultiLabel Relationship Learning (MLRL), which extends the conventional support vector machine by explicitly learning and utilizing the relationships between labels. Specifically, we model the label relationships using a label covariance matrix and use it to define a new regularization term for the optimization problem. MLRL learns the model parameters and the label covariance matrix simultaneously based on a unified convex formulation. To solve the convex optimization problem, we use an alternating method in which each subproblem can be solved efficiently. The relationship between MLRL and two widely used maximum margin methods for multilabel learning is investigated. Moreover, we also propose a semisupervised extension of MLRL, called SSMLRL, to demonstrate how to make use of unlabeled data to help learn the label covariance matrix. Through experiments conducted on some multilabel applications, we find that MLRL not only gives higher classification accuracy but also has better interpretability as revealed by the label covariance matrix.

Semi-Supervised Novelty Detection Using SVM Entire Solution Path

Very often, the only reliable information available to perform change detection is the description of some “unchanged” regions. Since, sometimes, these regions do not contain all the relevant information to identify their counterpart (the changes), we consider the use of unlabeled data to perform semi-supervised novelty detection (SSND). SSND can be seen as an unbalanced classification problem solved using the cost-sensitive support vector machine (CS-SVM), but this requires a heavy parameter search. Here, we propose the use of entire solution path algorithms for the CS-SVM in order to facilitate and accelerate parameter selection for SSND. Two algorithms are considered and evaluated. The first algorithm is an extension of the CS-SVM algorithm that returns the entire solution path in a single optimization. This way, optimization of a separate model for each hyperparameter set is avoided. The second algorithm forces the solution to be coherent through the solution path, thus producing classification boundaries that are nested (included in each other). We also present a low-density (LD) criterion for selecting optimal classification boundaries, thus avoiding recourse to cross validation (CV) that usually requires information about the “change” class. Experiments are performed on two multitemporal change detection data sets (flood and fire detection). Both algorithms tracing the solution path provide similar performances than the standard CS-SVM while being significantly faster. The proposed LD criterion achieves results that are close to the ones obtained by CV but without using information about the changes.

Cross-Lingual Adaptation Using Structural Correspondence Learning

Cross-lingual adaptation is a special case of domain adaptation and refers to the transfer of classification knowledge between two languages. In this article we describe an extension of Structural Correspondence Learning (SCL), a recently proposed algorithm for domain adaptation, for cross-lingual adaptation in the context of text classification. The proposed method uses unlabeled documents from both languages, along with a word translation oracle, to induce a cross-lingual representation that enables the transfer of classification knowledge from the source to the target language. The main advantages of this method over existing methods are resource efficiency and task specificity. We conduct experiments in the area of cross-language topic and sentiment classification involving English as source language and German, French, and Japanese as target languages. The results show a significant improvement of the proposed method over a machine translation baseline, reducing the relative error due to cross-lingual adaptation by an average of 30% (topic classification) and 59% (sentiment classification). We further report on empirical analyses that reveal insights into the use of unlabeled data, the sensitivity with respect to important hyperparameters, and the nature of the induced cross-lingual word correspondences.

On multivariate calibration with unlabeled data

AbstractIn principal component regression (PCR) and partial least‐squares regression (PLSR), the use of unlabeled data, in addition to labeled data, helps stabilize the latent subspaces in the calibration step, typically leading to a lower prediction error. For using unlabeled data in PLSR, a non‐sequential approach based on optimal filtering (OF) has been proposed in the literature. In this work, a sequential version of the OF‐based PLSR and a PCA‐based PLSR (PLSR applied to PCA‐preprocessed data) are proposed. It is shown analytically that the sequential version of the OF‐based PLSR is equivalent to that of PCA‐based PLSR, which leads to a new interpretation of OF. Simulated and experimental data sets are used to point out the usefulness and pitfalls of using unlabeled data. Unlabeled data can replace labeled data to some extent, thereby leading to an economic benefit. However, in the presence of drift, the use of unlabeled data can result in an increase in prediction error compared to that obtained with a model based on labeled data alone. Copyright © 2011 John Wiley & Sons, Ltd.

Software defect detection with rocus

Software defect detection aims to automatically identify defective software modules for efficient software test in order to improve the quality of a software system. Although many machine learning methods have been successfully applied to the task, most of them fail to consider two practical yet important issues in software defect detection. First, it is rather difficult to collect a large amount of labeled training data for learning a well-performing model; second, in a software system there are usually much fewer defective modules than defect-free modules, so learning would have to be conducted over an imbalanced data set. In this paper, we address these two practical issues simultaneously by prcposing a novel semi-supervised learning approach named ROCUS. This method exploits the abundant unlabeled examples to improve the detection accuracy, as well as employs under-sampling to tackle the class-imbalance problem in the learning process. Experimental results of real-world software defect detection tasks show that ROCUS is effective for software defect cetection. Its performance is better than a semi-supervised learning method that ignores the class-imbalance nature of the task and a class-imbalance learning method that does not make effective use of unlabeled data.