Unlabeled Training Research Articles

Minimax-Optimal Bounds for Detectors Based on Estimated Prior Probabilities

In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The error or {\em risk} is considered as a function of the prior probabilities. We show that the risk function is locally Lipschitz in the vicinity of the true prior probabilities, and the error of detectors based on estimated prior probabilities depends on the behavior of the risk function in this locality. In general, we show that the error of detectors based on the Maximum Likelihood Estimate (MLE) of the prior probabilities converges to the Bayes error at a rate of $n^{-1/2}$, where $n$ is the number of training data. If the behavior of the risk function is more favorable, then detectors based on the MLE have errors converging to the corresponding Bayes errors at optimal rates of the form $n^{-(1+\alpha)/2}$, where $\alpha>0$ is a parameter governing the behavior of the risk function with a typical value $\alpha = 1$. The limit $\alpha \rightarrow \infty$ corresponds to a situation where the risk function is flat near the true probabilities, and thus insensitive to small errors in the MLE; in this case the error of the detector based on the MLE converges to the Bayes error exponentially fast with $n$. We show the bounds are achievable no matter given labeled or unlabeled training data and are minimax-optimal in labeled case.

ECG Analysis Using Multiple Instance Learning for Myocardial Infarction Detection

This paper presents a useful technique for totally automatic detection of myocardial infarction from patients' ECGs. Due to the large number of heartbeats constituting an ECG and the high cost of having all the heartbeats manually labeled, supervised learning techniques have achieved limited success in ECG classification. In this paper, we first discuss the rationale for applying multiple instance learning (MIL) to automated ECG classification and then propose a new MIL strategy called latent topic MIL, by which ECGs are mapped into a topic space defined by a number of topics identified over all the unlabeled training heartbeats and support vector machine is directly applied to the ECG-level topic vectors. Our experimental results on real ECG datasets from the PTB diagnostic database demonstrate that, compared with existing MIL and supervised learning algorithms, the proposed algorithm is able to automatically detect ECGs with myocardial ischemia without labeling any heartbeats. Moreover, it improves classification quality in terms of both sensitivity and specificity.

${\rm S}^{3}{\rm MKL}$: Scalable Semi-Supervised Multiple Kernel Learning for Real-World Image Applications

We study the visual learning models that could work efficiently with little ground-truth annotation and a mass of noisy unlabeled data for large scale Web image applications, following the subroutine of semi-supervised learning (SSL) that has been deeply investigated in various visual classification tasks. However, most previous SSL approaches are not able to incorporate multiple descriptions for enhancing the model capacity. Furthermore, sample selection on unlabeled data was not advocated in previous studies, which may lead to unpredictable risk brought by real-world noisy data corpse. We propose a learning strategy for solving these two problems. As a core contribution, we propose a scalable semi-supervised multiple kernel learning method <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$({\rm S}^{3}{\rm MKL})$</tex></formula> to deal with the first problem. The aim is to minimize an overall objective function composed of log-likelihood empirical loss, conditional expectation consensus (CEC) on the unlabeled data and group LASSO regularization on model coefficients. We further adapt CEC into a group-wise formulation so as to better deal with the intrinsic visual property of real-world images. We propose a fast block coordinate gradient descent method with several acceleration techniques for model solution. Compared with previous approaches, our model better makes use of large scale unlabeled images with multiple feature representation with lower time complexity. Moreover, to address the issue of reducing the risk of using unlabeled data, we design a multiple kernel hashing scheme to identify the “informative” and “compact” unlabeled training data subset. Comprehensive experiments are conducted and the results show that the proposed learning framework provides promising power for real-world image applications, such as image categorization and personalized Web image re-ranking with very little user interaction.

Time Series Analysis Using Geometric Template Matching

We present a novel framework for analyzing univariate time series data. At the heart of the approach is a versatile algorithm for measuring the similarity of two segments of time series called geometric template matching (GeTeM). First, we use GeTeM to compute a similarity measure for clustering and nearest-neighbor classification. Next, we present a semi-supervised learning algorithm that uses the similarity measure with hierarchical clustering in order to improve classification performance when unlabeled training data are available. Finally, we present a boosting framework called TDEBOOST, which uses an ensemble of GeTeM classifiers. TDEBOOST augments the traditional boosting approach with an additional step in which the features used as inputs to the classifier are adapted at each step to improve the training error. We empirically evaluate the proposed approaches on several datasets, such as accelerometer data collected from wearable sensors and ECG data.

Efficient recurrent local search strategies for semi- and unsupervised regularized least-squares classification

Binary classification tasks are among the most important ones in the field of machine learning. One prominent approach to address such tasks are support vector machines which aim at finding a hyperplane separating two classes well such that the induced distance between the hyperplane and the patterns is maximized. In general, sufficient labeled data is needed for such classification settings to obtain reasonable models. However, labeled data is often rare in real-world learning scenarios while unlabeled data can be obtained easily. For this reason, the concept of support vector machines has also been extended to semi- and unsupervised settings: in the unsupervised case, one aims at finding a partition of the data into two classes such that a subsequent application of a support vector machine leads to the best overall result. Similarly, given both a labeled and an unlabeled part, semi-supervised support vector machines favor decision hyperplanes that lie in a low density area induced by the unlabeled training patterns, while still considering the labeled part of the data. The associated optimization problems for both the semi- and unsupervised case, however, are of combinatorial nature and, hence, difficult to solve. In this work, we present efficient implementations of simple local search strategies for (variants of) the both cases that are based on matrix update schemes for the intermediate candidate solutions. We evaluate the performances of the resulting approaches on a variety of artificial and real-world data sets. The results indicate that our approaches can successfully incorporate unlabeled data. (The unsupervised case was originally proposed by Gieseke F, Pahikkala et al. (2009). The derivations presented in this work are new and comprehend the old ones (for the unsupervised setting) as a special case.)

Semi-Supervised Learning for Classification with Uncertainty

Support vector machine (SVM) is a general and powerful learning machine, which adopts supervised manner. However, for many practical machine learning and data mining applications, unlabeled training examples are readily available but labeled ones are very expensive to be obtained. Therefore, semi-supervised learning emerges as the times require. At present, the combination of SVM and semi-supervised learning (S3VM) has attracted more and more attentions. In general, S3VM deals with problems with small training sets and large working sets. When the training set is large relative to the working set, We propose a new SVM model to solve the above classification problem by introducing the fuzzy memberships to each unlabeled point. Simulation results demonstrate that the proposed method can exploit unlabeled data to yield good performance effectively.

Video Behaviour Mining Using a Dynamic Topic Model

This paper addresses the problem of fully automated mining of public space video data, a highly desirable capability under contemporary commercial and security considerations. This task is especially challenging due to the complexity of the object behaviors to be profiled, the difficulty of analysis under the visual occlusions and ambiguities common in public space video, and the computational challenge of doing so in real-time. We address these issues by introducing a new dynamic topic model, termed a Markov Clustering Topic Model (MCTM). The MCTM builds on existing dynamic Bayesian network models and Bayesian topic models, and overcomes their drawbacks on sensitivity, robustness and efficiency. Specifically, our model profiles complex dynamic scenes by robustly clustering visual events into activities and these activities into global behaviours with temporal dynamics. A Gibbs sampler is derived for offline learning with unlabeled training data and a new approximation to online Bayesian inference is formulated to enable dynamic scene understanding and behaviour mining in new video data online in real-time. The strength of this model is demonstrated by unsupervised learning of dynamic scene models for four complex and crowded public scenes, and successful mining of behaviors and detection of salient events in each.

ROBUST CO-TRAINING

Co-training is a multiview semi-supervised learning algorithm to learn from both labeled and unlabeled data, which iteratively adopts a classifier trained on one view to teach the other view using some confident predictions given on unlabeled examples. However, as it does not examine the reliability of the labels provided by classifiers on either view, co-training might be problematic. Even very few inaccurately labeled examples can deteriorate the performance of learned classifiers to a large extent. In this paper, a new method named robust co-training is proposed, which integrates canonical correlation analysis (CCA) to inspect the predictions of co-training on those unlabeled training examples. CCA is applied to obtain a low-dimensional and closely correlated representation of the original multiview data. Based on this representation the similarities between an unlabeled example and the original labeled examples are determined. Only those examples whose predicted labels are consistent with the outcome of CCA examination are eligible to augment the original labeled data. The performance of robust co-training is evaluated on several different classification problems where encouraging experimental results are observed.

Performance of PCA Based Semi-supervised Learning in Face Recognition Using MPEG-7 Edge Histogram Descriptor

With the development of computer technology, the growing need of large databases emphasizes utmost automation in face recognition system. Labeling a huge number of people is definitely a great overhead for such system. To reduce the burden of manual labeling of faces, in this paper, we propose a PCA based semi-supervised face recognition technique using edge histogram descriptor (EHD) features. It promotes automation of system by achieving good performance toiling at minimum wage. Moreover, we establish that EHD features are not only useful for face representation but also greatly reduce the dimensionality of the representation compared with traditional pixel value representation. Thus, the space and computation complexity decrease in further stages. PCA is then applied on the EHD features instead of raw pixel intensity values of faces which traditional methods do. Using this process, we build a PCA based classifier that can iteratively update itself after classifying unlabeled training instances. We check the performance of the system using three different similarity measures. We also test our system with different levels of noise to simulate practical environment. To evaluate the proposed method, we have used ORL, Yale, Grimace and JAFFE face databases and achieve superior performance.

Breast Tissue Image Classification Based on Semi-supervised Locality Discriminant Projection with Kernels

Breast tissue classification is an important and effective way for computer aided diagnosis of breast cancer. We present Semi-supervised Locality Discriminant Projections with Kernels for breast cancer classification. The contributions of this work lie in: 1) Semi-supervised learning is used into Locality Preserving Projections (LPP) to enhance its performance using side-information together with the unlabelled training samples, while current algorithms only consider the side-information but ignoring the unlabeled training samples. 2) Kernel trick is applied into Semi-supervised LPP to improve its ability in the nonlinear classification. 3) The framework of breast cancer classification with Semi-supervised LPP with kernels is presented. Many experiments are implemented on four breast tissue databases to testify and evaluate the feasibility and affectivity of the proposed scheme.

A Co-Training Based Semi-Supervised Human Action Recognition Algorithm

A novel semi-supervised algorithm based on co-training is proposed in this paper. In the method, the motion energy history image are used as the different feature representation of human action; then the co-training based semi-supervised learning algorithm is utilized to predict the category of unlabeled training examples. And the average motion energy and history images are calculated as the recognition model for each category action. When recognition, the observed action is firstly classified through its correlation coefficients to the prior established templates respectively; then its final category is determined according to the consistency between the classification results of motion energy and motion history images. The experiments on Weizmann dataset demonstrate that our method is effective for human action recognition.

Optional SVM for Fault Diagnosis of Blast Furnace with Imbalanced Data

Fault diagnosis for blast furnace is actually a multi-class classification problem because the blast furnace may appear usually many kinds of abnormal states. Moreover, those abnormal states should be monitored and diagnosed timely and what can help workers take effective measures. Support vector machine (SVM) is state-of-the-art for many classification problems currently. But many classification tasks involve imbalanced training examples in practice. Imbalanced dataset learning is an important practical issue in machine learning, especially in support vector machine (SVM). Fault diagnosis for blast furnace is such an imbalanced data problem. A novel algorithm named optional support vector machine is proposed to solve this imbalanced data classification by pruning training sets and adding the unlabeled data and applying edited nearest neighbor (ENN) rules. Firstly, training sets of majority class are pruned in order to reduce the training time. Secondly, the algorithm selects some useful unlabelled training data and adds them to the training sets. Those samples are used to replenish the lack of training samples so that the training sets are representative. However, they may contain some noisy examples. Finally, the edited nearest neighbor rule is removed the noisy examples. The algorithm adds the unlabelled (testing) samples to balance the number of samples between the minority class and the majority one. The real-time producing data of blast furnace are used to running experiment. In order to more accurately diagnose which kinds fault happened, a binary tree multi-class classification method is adopted based on blast furnace characteristics. Simulation results show that the proposed algorithm is feasible and effective.

A hybrid classification scheme for mining multisource geospatial data

Supervised learning methods such as Maximum Likelihood (ML) are often used in land cover (thematic) classification of remote sensing imagery. ML classifier relies exclusively on spectral characteristics of thematic classes whose statistical distributions (class conditional probability densities) are often overlapping. The spectral response distributions of thematic classes are dependent on many factors including elevation, soil types, and ecological zones. A second problem with statistical classifiers is the requirement of the large number of accurate training samples (10 to 30 × |dimensions|), which are often costly and time consuming to acquire over large geographic regions. With the increasing availability of geospatial databases, it is possible to exploit the knowledge derived from these ancillary datasets to improve classification accuracies even when the class distributions are highly overlapping. Likewise newer semi-supervised techniques can be adopted to improve the parameter estimates of the statistical model by utilizing a large number of easily available unlabeled training samples. Unfortunately, there is no convenient multivariate statistical model that can be employed for multisource geospatial databases. In this paper we present a hybrid semi-supervised learning algorithm that effectively exploits freely available unlabeled training samples from multispectral remote sensing images and also incorporates ancillary geospatial databases. We have conducted several experiments on Landsat satellite image datasets, and our new hybrid approach shows over 24% to 36% improvement in overall classification accuracy over conventional classification schemes.

Tutor-based learning of visual categories using different levels of supervision

In recent years we have seen lots of strong work in visual recognition, dialogue interpretation and multi-modal learning that is targeted at provide the building blocks to enable intelligent robots to interact with humans in a meaningful way and even continuously evolve during this process. Building systems that unify those components under a common architecture has turned out to be challenging, as each approach comes with it’s own set of assumptions, restrictions, and implications. For example, the impact of recent progress on visual category recognition has been limited from a perspective of interactive systems. Reasons for this are diverse. We identify and address two major challenges in order to integrate modern techniques for visual categorization in an interactive learning system: reducing the number of required labelled training examples and dealing with potentially erroneous input. Today’s object categorization methods use either supervised or unsupervised training methods. While supervised methods tend to produce more accurate results, unsupervised methods are highly attractive due to their potential to use far more and unlabeled training data. We proposes a novel method that uses unsupervised training to obtain visual groupings of objects and a cross-modal learning scheme to overcome inherent limitations of purely unsupervised training. The method uses a unified and scale-invariant object representation that allows to handle labeled as well as unlabeled information in a coherent way. First experiments demonstrate the ability of the system to learn object category models from many unlabeled observations and a few dialogue interactions that can be ambiguous or even erroneous.

Sparsity preserving discriminant analysis for single training image face recognition

Single training image face recognition is one of the main challenges to appearance-based pattern recognition techniques. Many classical dimensionality reduction methods such as LDA have achieved success in face recognition field, but cannot be directly used to the single training image scenario. Recent graph-based semi-supervised dimensionality reduction (SSDR) provides a feasible strategy to deal with such problem. However, most of the existing SSDR algorithms such as semi-supervised discriminant analysis (SDA) are locality-oriented and generally suffer from the following issues: (1) they need a large number of unlabeled training samples to estimate the manifold structure in data, but such extra samples may not be easily obtained in a given face recognition task; (2) they model the local geometry of data by the nearest neighbor criterion which generally fails to obtain sufficient discriminative information due to the high-dimensionality of face image space; (3) they construct the underlying adjacency graph (or data-dependent regularizer) using a fixed neighborhood size for all the sample points without considering the actual data distribution. In this paper, we develop a new graph-based SSDR algorithm called sparsity preserving discriminant analysis (SPDA) to address these problems. More specifically, (1) the graph in SPDA is constructed by sparse representation, and thus the local structure in data is automatically modeled instead of being manually predefined. (2) With the natural discriminative power of sparse representation, SPDA can remarkably improve recognition performance only resorting to very few extra unlabeled samples. (3) A simple ensemble strategy is developed to accelerate graph construction, which results in an efficient ensemble SPDA algorithm. Extensive experiments on both toy and real face data sets are provided to validate the feasibility and effectiveness of the proposed algorithm.

Hypergraph-Based Anomaly Detection of High-Dimensional Co-Occurrences

This paper addresses the problem of detecting anomalous multivariate co-occurrences using a limited number of unlabeled training observations. A novel method based on using a hypergraph representation of the data is proposed to deal with this very high-dimensional problem. Hypergraphs constitute an important extension of graphs which allow edges to connect more than two vertices simultaneously. A variational Expectation-Maximization algorithm for detecting anomalies directly on the hypergraph domain without any feature selection or dimensionality reduction is presented. The resulting estimate can be used to calculate a measure of anomalousness based on the False Discovery Rate. The algorithm has O(np) computational complexity, where n is the number of training observations and p is the number of potential participants in each co-occurrence event. This efficiency makes the method ideally suited for very high-dimensional settings, and requires no tuning, bandwidth or regularization parameters. The proposed approach is validated on both high-dimensional synthetic data and the Enron email database, where p > 75,000, and it is shown that it can outperform other state-of-the-art methods.

Semi-supervised Bayesian ARTMAP

This paper proposes a semi-supervised Bayesian ARTMAP (SSBA) which integrates the advantages of both Bayesian ARTMAP (BA) and Expectation Maximization (EM) algorithm. SSBA adopts the training framework of BA, which makes SSBA adaptively generate categories to represent the distribution of both labeled and unlabeled training samples without any user's intervention. In addition, SSBA employs EM algorithm to adjust its parameters, which realizes the soft assignment of training samples to categories instead of the hard assignment such as winner takes all. Experimental results on benchmark and real world data sets indicate that the proposed SSBA achieves significantly improved performance compared with BA and EM-based semi-supervised learning method; SSBA is appropriate for semi-supervised classification tasks with large amount of unlabeled samples or with strict demands for classification accuracy.

Audio-visual human recognition using semi-supervised spectral learning and hidden Markov models

This paper presents a multimodal system for reliable human identity recognition under variant conditions. Our system fuses the recognition of face and speech with a general probabilistic framework. For face recognition, we propose a new spectral learning algorithm, which considers not only the discriminative relations among the training data but also the generative models for each class. Due to the tedious cost of face labeling in practice, our spectral face learning utilizes a semi-supervised strategy. That is, only a small number of labeled faces are used in our training step, and the labels are optimally propagated to other unlabeled training faces. Besides requiring much less labeled data, our algorithm also enables a natural way to explicitly train an outlier model that approximately represents unauthorized faces. To boost the robustness of our system for human recognition under various environments, our face recognition is further complemented by a speaker identification agent. Specifically, this agent models the statistical variations of fixed-phrase speech using speaker-dependent word hidden Markov models. Experiments on benchmark databases validate the effectiveness of our face recognition and speaker identification agents, and demonstrate that the recognition accuracy can be apparently improved by integrating these two independent biometric sources together.

Exploiting unlabeled data to improve peer-to-peer traffic classification using incremental tri-training method

Unlabeled training examples are readily available in many applications, but labeled examples are fairly expensive to obtain. For instance, in our previous works on classification of peer-to-peer (P2P) Internet traffics, we observed that only about 25% of examples can be labeled as “P2P”or “NonP2P” using a port-based heuristic rule. We also expect that even fewer examples can be labeled in the future as more and more P2P applications use dynamic ports. This fact motivates us to investigate the techniques which enhance the accuracy of P2P traffic classification by exploiting the unlabeled examples. In addition, the Internet data flows dynamically in large volumes (streaming data). In P2P applications, new communities of peers often join and old communities of peers often leave, requiring the classifiers to be capable of updating the model incrementally, and dealing with concept drift. Based on these requirements, this paper proposes an incremental Tri-Training (iTT) algorithm. We tested our approach on a real data stream with 7.2 Mega labeled examples and 20.4 Mega unlabeled examples. The results show that iTT algorithm can enhance accuracy of P2P traffic classification by exploiting unlabeled examples. In addition, it can effectively deal with dynamic nature of streaming data to detect the changes in communities of peers. We extracted attributes only from the IP layer, eliminating the privacy concern associated with the techniques that use deep packet inspection.

LOW-EFFORT LABELING OF NETWORK EVENTS FOR INTRUSION DETECTION IN WLANS

A typical data mining approach to network intrusion detection mandates a training dataset of network events labeled as either normal or a particular attack category. Such a training dataset is usually very large since there are many events to track. This is particularly the case in a WLAN where the number of devices communicating with the WLAN can be large and with adhoc connectivity. The large size of the unlabeled training dataset creates a problem for the domain expert who is asked to label the records toward creating a training dataset. We present an effective approach by which the number of network records the expert has to examine is a relatively small proportion of the given training dataset. A clustering algorithm is used to form relatively coherent groups which the expert examines as an entity to label records as one of four classes: Red (definite intrusion), Yellow (possibly intrusion), Blue (probably normal), and Green (definite normal). Subsequently, an ensemble classifier-based data cleansing approach is used to detect records that were likely mislabeled by the expert. The proposed approach is investigated with a case study of a large real-world WLAN. In addition, ensemble classifier-based intrusion detection models built using the labeled training dataset demonstrate the effectiveness of the labeling process with good generalization accuracy over multiple test datasets.