Self-training Algorithm Research Articles

Image Clustering Based on Multi-Scale Deep Maximize Mutual Information and Self-Training Algorithm

Image clustering is a complex procedure that is significantly affected by the choice of the image representation. Generally speaking, image representations are generated by using handcraft features or trained neural networks. When dealing with high dimension data, these two representation methods cause two problems: i) the representation ability of the manually designed features is limited; ii) the non-representative and meaningless feature of a trained deep network may hurt the clustering performance. To overcome these problems, we propose a new clustering method which efficiently builds an image representation and precisely discovers the cluster assignments. Our main tools are an unsupervised representation learning method based on Deep Mutual Information Maximization (DMIM) system, and a clustering method based on self-training algorithm. Specifically speaking, to extract the informative representation of image data, we derive the maximum mutual information theory and propose a system to learn the maximum mutual information between the input images and the latent representations. To discover the clusters and assign each image a clustering label, a self-training mechanism is applied to cluster the learned representations. The superiority and validity of our algorithm are verified in a series of real-world image clustering experiments.

Semi-Supervised Self-Training Feature Weighted Clustering Decision Tree and Random Forest

A self-training algorithm is an iterative method for semi-supervised learning, which wraps around a base learner. It uses its own predictions to assign labels to unlabeled data. For a self-training algorithm, the classification ability of the base learner and the estimation of prediction confidence are very important. The classical decision tree as the base learner cannot be effective in a self-training algorithm, because it cannot correctly estimate its own predictions. In this paper, we propose a novel method of node split of the decision trees, which uses weighted features to cluster instances. This method is able to combine multiple numerical and categorical features to split nodes. The decision tree and random forest constructed by this method are called FWCDT and FWCRF respectively. FWCDT and FWCRF have the better classification ability than the classical decision trees and forests based on univariate split when the training instances are fewer, therefore, they are more suitable as the base classifiers in self-training. What’s more, on the basis of the proposed node-split method, we also respectively explore the suitable prediction confidence measurements for FWCDT and FWCRF. Finally, the results of experiment implemented on the UCI datasets show that the self-training feature weighted clustering decision tree (ST-FWCDT) and random forest (ST-FWCRF) can effectively exploit unlabeled data, and the final obtained classifiers have better generalization ability.

Deep Contextualized Self-training for Low Resource Dependency Parsing

Neural dependency parsing has proven very effective, achieving state-of-the-art results on numerous domains and languages. Unfortunately, it requires large amounts of labeled data, which is costly and laborious to create. In this paper we propose a self-training algorithm that alleviates this annotation bottleneck by training a parser on its own output. Our Deep Contextualized Self-training (DCST) algorithm utilizes representation models trained on sequence labeling tasks that are derived from the parser’s output when applied to unlabeled data, and integrates these models with the base parser through a gating mechanism. We conduct experiments across multiple languages, both in low resource in-domain and in cross-domain setups, and demonstrate that DCST substantially outperforms traditional self-training as well as recent semi-supervised training methods. 1

A semi-supervised self-trained two-level algorithm for forecasting students’ graduation time

During the last decades, educational data mining has become a significant tool for the prediction of students’ progress and performance. In this work, we present a new semi-supervised self-trained two-level classification algorithm for predicting students’ graduation time. The proposed algorithm ha s three major features: Firstly, it identifies with high accuracy the students at-risk of not completing their studies; secondly, it classifies the students based on their expected graduation time; thirdly, it meaningfully relates the explicit classification information of labeled data with the information hidden in the unlabeled data. Our preliminary numerical experiments indicate that the proposed algorithm exhibits reliable predictions based on the students’ performance during the first two years of their studies.

LabelForest: Non-Parametric Semi-Supervised Learning for Activity Recognition

Activity recognition is central to many motion analysis applications ranging from health assessment to gaming. However, the need for obtaining sufficiently large amounts of labeled data has limited the development of personalized activity recognition models. Semi-supervised learning has traditionally been a promising approach in many application domains to alleviate reliance on large amounts of labeled data by learning the label information from a small set of seed labels. Nonetheless, existing approaches perform poorly in highly dynamic settings, such as wearable systems, because some algorithms rely on predefined hyper-parameters or distribution models that needs to be tuned for each user or context. To address these challenges, we introduce LabelForest 1, a novel non-parametric semi-supervised learning framework for activity recognition. LabelForest has two algorithms at its core: (1) a spanning forest algorithm for sample selection and label inference; and (2) a silhouette-based filtering method to finalize label augmentation for machine learning model training. Our thorough analysis on three human activity datasets demonstrate that LabelForest achieves a labeling accuracy of 90.1% in presence of a skewed label distribution in the seed data. Compared to self-training and other sequential learning algorithms, LabelForest achieves up to 56.9% and 175.3% improvement in the accuracy on balanced and unbalanced seed data, respectively.

Diversity-Based Random Forests with Sample Weight Learning

Given a variety of classifiers, one prevalent approach in classifier ensemble is to diversely combine classifier components, i.e., diversity-based ensembles, and a lot of previous works show that these ensembles can improve classification accuracy. Random forests are one of the most important ensembles. However, most random forests approaches with diversity-related aspects focus on maximizing tree diversity while producing and training component trees. Alternatively, a novel cognitive-inspired diversity-based random forests method, diversity-based random forests via sample weight learning (DRFS), is proposed. Given numerous component trees from the original random forests, DRFS selects and combines tree classifiers adaptively via diversity learning and sample weight learning. By designing a matrix for the data distribution creatively, a unified optimization model is formulated to learn and select diverse trees, where tree weights are learned through a convex quadratic programming problem with sample weights. Moreover, a self-training algorithm is proposed to solve the convex optimization iteratively and learn sample weights automatically. Comparative experiments on 39 typical UCI classification benchmarks and a variety of real-world text categorization benchmarks of our proposed method are conducted. Extensive experiments show that our method outperforms the traditional methods. Our proposed DRFS method can select and combine tree classifiers adaptively and improves the performance on a variety of classification tasks.

An Auto-Adjustable Semi-Supervised Self-Training Algorithm

Semi-supervised learning algorithms have become a topic of significant research as an alternative to traditional classification methods which exhibit remarkable performance over labeled data but lack the ability to be applied on large amounts of unlabeled data. In this work, we propose a new semi-supervised learning algorithm that dynamically selects the most promising learner for a classification problem from a pool of classifiers based on a self-training philosophy. Our experimental results illustrate that the proposed algorithm outperforms its component semi-supervised learning algorithms in terms of accuracy, leading to more efficient, stable and robust predictive models.

ST-IRGS: A Region-Based Self-Training Algorithm Applied to Hyperspectral Image Classification and Segmentation

The problem of limited labeled training samples is challenging for the classification of remote sensing imagery. We develop a joint classification and segmentation algorithm to address this problem. Our algorithm combines semisupervised learning and conditional random fields (CRFs) into a single framework. The multimodal Gaussian maximum-likelihood classifier is used to estimate the probabilities for the unary potentials of the CRF. Unlike traditional methods based on random fields, region merging is concatenated with the CRF inference to reduce the number of nodes iteratively. Moreover, a semisupervised technique called self-training is used, which iteratively enlarges the training sample set and retrains the classifier. The selection of training samples is based on the region information, so that the risk of assigning wrong labels is largely reduced. The proposed algorithm is applied to hyperspectral image classification, and results on benchmark data sets show that the proposed algorithm significantly improves classification performance after using self-training, and outperforms state-of-the-art spectral–spatial methods for limited labeled training samples.

Multiple kernel approach to semi-supervised fuzzy clustering algorithm for land-cover classification

Clustering is used to detect sound structures or patterns in a dataset in which objects positioned within the same cluster exhibit a substantial level of similarity. In numerous clustering problems, patterns is not easily separable due to the highly complex shaped data. In the previous studies, kernel-based methods have exhibited the effectiveness to partition such data. In this paper, we proposed a semi-supervised clustering method based fuzzy c-means algorithm using multiple kernel technique, called SMKFCM, in which the rudimentary centroids are directly used to the calculating process of centroids. The SMKFCM algorithm is on the basis of combining the labeled and unlabeled data together to improve performance. We used the labeled patterns to calculate the centrality of clusters considered as the rudimentary centroids which are added into the objective functions. The SMKFCM algorithm can be applied to both clustering and classification problems. The experimental results show that SMKFCM algorithm can improve significantly the classification accuracy which comes from comparison with a conventional classification or clustering algorithms such as semi-supervised kernel fuzzy c-means (S2KFCM), semi-supervised fuzzy c-means (SFCM) and Self-trained semi-supervised SVM algorithm (PS3VM).

A Self-Training Subspace Clustering Algorithm under Low-Rank Representation for Cancer Classification on Gene Expression Data.

Accurate identification of the cancer types is essential to cancer diagnoses and treatments. Since cancer tissue and normal tissue have different gene expression, gene expression data can be used as an efficient feature source for cancer classification. However, accurate cancer classification directly using original gene expression profiles remains challenging due to the intrinsic high-dimension feature and the small size of the data samples. We proposed a new self-training subspace clustering algorithm under low-rank representation, called SSC-LRR, for cancer classification on gene expression data. Low-rank representation (LRR) is first applied to extract discriminative features from the high-dimensional gene expression data; the self-training subspace clustering (SSC) method is then used to generate the cancer classification predictions. The SSC-LRR was tested on two separate benchmark datasets in control with four state-of-the-art classification methods. It generated cancer classification predictions with an overall accuracy 89.7 percent and a general correlation 0.920, which are 18.9 and 24.4 percent higher than that of the best control method respectively. In addition, several genes (RNF114, HLA-DRB5, USP9Y, and PTPN20) were identified by SSC-LRR as new cancer identifiers that deserve further clinical investigation. Overall, the study demonstrated a new sensitive avenue to recognize cancer classifications from large-scale gene expression data.

An Improved Self-Training Algorithm for Classifying Motor Imagery Electroencephalography in Brain-Computer Interface

An Improved Self-Training Algorithm for Classifying Motor Imagery Electroencephalography in Brain-Computer Interface

Reinforced confidence in self-training for a semi-supervised medical data classification

The implication of semi-supervised method has become crucial to automate tasks that require the manual human expertise for data labelling, the advantage of this method resides in the fact that they require a low amount of labelled information. In this work, we are particularly interested in self-training paradigm. These techniques use the same principle as those of supervised techniques, but with a confidence measure that allows only a selection of the most confident samples. We propose a novel self-training algorithm named reinforced confidence in self-training (R-COSET) based on an iterative process. In each iteration the learned hypothesis can be improved by confidence data, where the proposed confidence measure is reinforced by two confidence levels in order to increase the robustness of the self-training process. Experiments show that the introduction of the second level of the neighbourhood graph in confidence measure is beneficial and that R-COSET can effectively improve classification performance.

Improving semi-supervised self-training with embedded manifold transduction

Semi-supervised learning aims to utilize both labelled and unlabelled data to improve learning performance. This paper shows a distinct way to exploit unlabelled data for traditional semi-supervised learning methods, such as self-training. Self-training is a well-known semi-supervised learning algorithm which iteratively trains a classifier by bootstrapping from unlabelled data. Standard self-training barely selects unlabelled examples for training set augmentation according to the current classifier model, which is trained only on the labelled data. This could be problematic since the underlying classifier is not strong enough, especially when initial labelled data is sparse. Consequently, self-training suffers from too much classification noise accumulated in the training set. In this paper, we propose a novel self-training style algorithm, which exploits a manifold assumption to optimize the self-labelling process. Unlike standard self-training, our algorithm utilizes labelled and unlabelled data as a whole to label and select unlabelled examples for training set augmentation. In detail, two measures are employed to minimize the effect of noise introduced to the labelled training set: a transductive method based on controlled graph random walk is incorporated to generate reliable predictions on unlabelled data; secondly, the mechanism is adopted to sequentially augment the training set. Empirical results suggest that the proposed method can effectively improve classification performance.

Predicting pupylation sites in prokaryotic proteins using semi-supervised self-training support vector machine algorithm

As one important post-translational modification of prokaryotic proteins, pupylation plays a key role in regulating various biological processes. The accurate identification of pupylation sites is crucial for understanding the underlying mechanisms of pupylation. Although several computational methods have been developed for the identification of pupylation sites, the prediction accuracy of them is still unsatisfactory. Here, a novel bioinformatics tool named IMP–PUP is proposed to improve the prediction of pupylation sites. IMP–PUP is constructed on the composition of k-spaced amino acid pairs and trained with a modified semi-supervised self-training support vector machine (SVM) algorithm. The proposed algorithm iteratively trains a series of support vector machine classifiers on both annotated and non-annotated pupylated proteins. Computational results show that IMP–PUP achieves the area under receiver operating characteristic curves of 0.91, 0.73, and 0.75 on our training set, Tung's testing set, and our testing set, respectively, which are better than those of the different error costs SVM algorithm and the original self-training SVM algorithm. Independent tests also show that IMP–PUP significantly outperforms three other existing pupylation site predictors: GPS–PUP, iPUP, and pbPUP. Therefore, IMP–PUP can be a useful tool for accurate prediction of pupylation sites. A MATLAB software package for IMP–PUP is available at https://juzhe1120.github.io/.

Using unsupervised information to improve semi-supervised tweet sentiment classification

Supervised algorithms require a set of representative labeled data for building classification models. However, labeled data are usually difficult and expensive to obtain, which motivates the interest in semi-supervised learning. This type of learning uses both labeled and unlabeled data in the training process and is particularly useful in applications such as tweet sentiment analysis, where a large amount of unlabeled data is available. Semi-supervised learning for tweet sentiment analysis, although quite appealing, is relatively new. We propose a semi-supervised learning framework that combines unsupervised information, captured from a similarity matrix constructed from unlabeled data, with a classifier. Our motivation is that such a similarity matrix is a powerful knowledge-discovery tool that can help classify unlabeled tweet sets. Our framework makes use of the well-known Self-training algorithm to induce a better tweet sentiment classifier. Experimental results in real-world datasets demonstrate that the proposed framework can improve the accuracy of tweet sentiment analysis.

An iterative self-training support vector machine algorithm in brain-computer interfaces

In this paper, an iterative self-training Support Vector Machine (SVM) algorithm combined feature re-extraction is proposed for semi-supervised learning, which only needs a small set of labeled samples to train classifier and is thus very useful in Brain-Computer Interface (BCI) design. Two methods, the model selection based self-training and the confidence criterion, respectively, is also proposed for searching the best parameter pair of SVM and selecting the most useful unlabeled data to expand the labeled training data set. The Dataset IVa of BCI Competition III, is presented to demonstrate the validity of our algorithm with statistical significance test. As an iterative algorithm, experimental results of the proposed algorithm show the validity of re-extracting feature and the robustness of the feature to the noise. In addition, the convergence of the proposed algorithm and the validity of the method measuring the consistency of the feature are also demonstrated in experiments.

Predicting alternatively spliced exons using semi-supervised learning

Cost-efficient next generation sequencers can now produce unprecedented volumes of raw DNA data, posing challenges for annotation. Supervised machine learning approaches have been traditionally used to analyse and annotate complex genomic information. However, such approaches require labelled data for training, which in practice is scarce or expensive, while the unlabelled data is abundant. For some problems, semi-supervised learning can help improve supervised classifiers by making use of large amounts of unlabelled data and the latent information within them. We evaluate the applicability of semi-supervised learning algorithms to the problem of DNA sequence annotation, specifically to the prediction of alternatively spliced exons. We employ Expectation Maximisation, Self-training, and Co-training algorithms in an effort to assess the strengths and limitations of these techniques in the context of alternative splicing.

An ant colony-based semi-supervised approach for learning classification rules

Semi-supervised learning methods create models from a few labeled instances and a great number of unlabeled instances. They appear as a good option in scenarios where there is a lot of unlabeled data and the process of labeling instances is expensive, such as those where most Web applications stand. This paper proposes a semi-supervised self-training algorithm called Ant-Labeler. Self-training algorithms take advantage of supervised learning algorithms to iteratively learn a model from the labeled instances and then use this model to classify unlabeled instances. The instances that receive labels with high confidence are moved from the unlabeled to the labeled set, and this process is repeated until a stopping criteria is met, such as labeling all unlabeled instances. Ant-Labeler uses an ACO algorithm as the supervised learning method in the self-training procedure to generate interpretable rule-based models—used as an ensemble to ensure accurate predictions. The pheromone matrix is reused across different executions of the ACO algorithm to avoid rebuilding the models from scratch every time the labeled set is updated. Results showed that the proposed algorithm obtains better predictive accuracy than three state-of-the-art algorithms in roughly half of the datasets on which it was tested, and the smaller the number of labeled instances, the better the Ant-Labeler performance.

Self-training in significance space of support vectors for imbalanced biomedical event data.

BackgroundPairwise relationships extracted from biomedical literature are insufficient in formulating biomolecular interactions. Extraction of complex relations (namely, biomedical events) has become the main focus of the text-mining community. However, there are two critical issues that are seldom dealt with by existing systems. First, an annotated corpus for training a prediction model is highly imbalanced. Second, supervised models trained on only a single annotated corpus can limit system performance. Fortunately, there is a large pool of unlabeled data containing much of the domain background that one can exploit.ResultsIn this study, we develop a new semi-supervised learning method to address the issues outlined above. The proposed algorithm efficiently exploits the unlabeled data to leverage system performance. We furthermore extend our algorithm to a two-phase learning framework. The first phase balances the training data for initial model induction. The second phase incorporates domain knowledge into the event extraction model. The effectiveness of our method is evaluated on the Genia event extraction corpus and a PubMed document pool. Our method can identify a small subset of the majority class, which is sufficient for building a well-generalized prediction model. It outperforms the traditional self-training algorithm in terms of f-measure. Our model, based on the training data and the unlabeled data pool, achieves comparable performance to the state-of-the-art systems that are trained on a larger annotated set consisting of training and evaluation data.

A co-training algorithm based on modified Fisher's linear discriminant analysis

In this paper, a new co-training algorithm based on modified Fisher's Linear Discriminant Analysis FLDA is proposed for semi-supervised learning, which only needs a small set of labeled samples to train classifiers and is thus very useful in applications like brain-computer interface BCI design. Two classifiers, one aiming to maximize the normalized between-class diversity and the other to minimize the normalized within-class diversity, are proposed for the co-training process. A method with a confidence criterion is also proposed for selecting unlabeled data to expand training data set. The co-training algorithm is compared with a static FLDA method and a FLDA based on self-training algorithm on the data set 2a for BCI Competition IV, with statistical significance test. Experimental results show that the new co-training algorithm outperformed the other two methods and its average classification accuracy was improved iteration by iteration, demonstrating the convergence of the co-training process.