Self-training Method Research Articles

Improving self-training with density peaks of data and cut edge weight statistic

Semi-supervised classification has become an active topic recently, and a number of algorithms, such as self-training, have been proposed to improve the performance of supervised classification using unlabeled data. Considering the influence of spatial distribution of data set and mislabeled samples on the classification performance of self-training method, an improved self-training algorithm based on density peaks and cut edge weight statistic is proposed in this paper. Firstly, the representative unlabeled samples are selected for labels prediction by space structure, which is discovered by clustering method based on density peaks. Secondly, cut edge weight is used as statistics to make hypothesis testing for identifying whether samples are labeled correctly. Thirdly, the labeled data set is gradually enlarged with correctly labeled samples. The above steps are iterated until all unlabeled samples are labeled. The framework of improved self-training method not only makes full use of space structure information, but also solves the problem that some samples may be classified incorrectly. Thus, the classification accuracy of algorithm is improved in a great measure. Extensive experiments on benchmark data sets clearly illustrate the effectiveness of proposed algorithm.

Unsupervised Domain Adaptation on Reading Comprehension

Reading comprehension (RC) has been studied in a variety of datasets with the boosted performance brought by deep neural networks. However, the generalization capability of these models across different domains remains unclear. To alleviate the problem, we investigate unsupervised domain adaptation on RC, wherein a model is trained on the labeled source domain and to be applied to the target domain with only unlabeled samples. We first show that even with the powerful BERT contextual representation, a model can not generalize well from one domain to another. To solve this, we provide a novel conditional adversarial self-training method (CASe). Specifically, our approach leverages a BERT model fine-tuned on the source dataset along with the confidence filtering to generate reliable pseudo-labeled samples in the target domain for self-training. On the other hand, it further reduces domain distribution discrepancy through conditional adversarial learning across domains. Extensive experiments show our approach achieves comparable performance to supervised models on multiple large-scale benchmark datasets.

Adversarial-Learned Loss for Domain Adaptation

Recently, remarkable progress has been made in learning transferable representation across domains. Previous works in domain adaptation are majorly based on two techniques: domain-adversarial learning and self-training. However, domain-adversarial learning only aligns feature distributions between domains but does not consider whether the target features are discriminative. On the other hand, self-training utilizes the model predictions to enhance the discrimination of target features, but it is unable to explicitly align domain distributions. In order to combine the strengths of these two methods, we propose a novel method called Adversarial-Learned Loss for Domain Adaptation (ALDA). We first analyze the pseudo-label method, a typical self-training method. Nevertheless, there is a gap between pseudo-labels and the ground truth, which can cause incorrect training. Thus we introduce the confusion matrix, which is learned through an adversarial manner in ALDA, to reduce the gap and align the feature distributions. Finally, a new loss function is auto-constructed from the learned confusion matrix, which serves as the loss for unlabeled target samples. Our ALDA outperforms state-of-the-art approaches in four standard domain adaptation datasets. Our code is available at https://github.com/ZJULearning/ALDA.

Abnormal vibration detection in the bearing-shaft system via semi-supervised classification of accelerometer signal patterns

Several methods have been proposed for fault detection in mechanical systems based on sensor signals. It is preferable that corresponding label for each sensor signal should be provided and analyzed via appropriate supervised classification methods. However, the label information about a system’s status often does not perfectly pair to the corresponding data. Therefore, we apply a semi-supervised classification for fault detection using pattern extraction of multivariate signals. This approach transforms continuous time series into a set of contiguous bins via multivariate discretization. Then, we identify informative patterns in the system states, by using a self-training method with limited label information. To demonstrate the effectiveness of the proposed extraction method, five accelerometer signals are collected from a bearing-shaft system. The proposed method successfully reveals informative fault patterns that can be applied as references for fault detection. The method achieved a higher detection performance, regardless the ratio of unlabeled inputs in datasets.

Semi-Supervised Learning for Fine-Grained Classification With Self-Training

Semi-supervised learning is a machine learning approach that tackles the challenge of having a large set of unlabeled data and few labeled ones. In this paper we adopt a semi-supervised self-training method to increase the amount of training data, prevent overfitting and improve the performance of deep models by proposing a novel selection algorithm that prevents mistake reinforcement which is a common thing in conventional self-training models. The model leverages, unlabeled data and specifically, after each training, we first generate pseudo-labels on the unlabeled set to be added to the labeled training samples. Next, we select the top-k most-confident pseudo-labeled images from each unlabeled class with their pseudo-labels and update the training data, and retrain the network on the updated training data. The method improves the accuracy in two-fold; bridging the gap in the appearance of visual objects, and enlarging the training set to meet the demands of deep models. We demonstrated the effectiveness of the model by conducting experiments on four state-of-the-art fine-grained datasets, which include Stanford Dogs, Stanford Cars, 102-Oxford flowers, and CUB-200-2011. We further evaluated the model on some coarse-grain data. Experimental results clearly show that our proposed framework has better performance than some previous works on the same data; the model obtained higher classification accuracy than most of the supervised learning models.

Improved well-log classification using semisupervised label propagation and self-training, with comparisons to popular supervised algorithms

Machine-learning techniques allow geoscientists to extract meaningful information from data in an automated fashion, and they are also an efficient alternative to traditional manual interpretation methods. Many geophysical problems have an abundance of unlabeled data and a paucity of labeled data, and the lithology classification of wireline data reflects this situation. Training supervised algorithms on small labeled data sets can lead to overtraining, and subsequent predictions for the numerous unlabeled data may be unstable. However, semisupervised algorithms are designed for classification problems with limited amounts of labeled data, and they are theoretically able to achieve better accuracies than supervised algorithms in these situations. We explore this hypothesis by applying two semisupervised techniques, label propagation (LP) and self-training, to a well-log data set and compare their performance to three popular supervised algorithms. LP is an established method, but our self-training method is a unique adaptation of existing implementations. The well-log data were made public through an SEG competition held in 2016. We simulate a semisupervised scenario with these data by assuming that only one of the 10 wells has labels (i.e., core samples), and our objective is to predict the labels for the remaining nine wells. We generate results from these data in two stages. The first stage is applying all the algorithms in question to the data as is (i.e., the global data), and the results from this motivate the second stage, which is applying all algorithms to the data when they are decomposed into two separate data sets. Overall, our findings suggest that LP does not outperform the supervised methods, but our self-training method coupled with LP can outperform the supervised methods by a notable margin if the assumptions of LP are met.

Self-Training Classification Framework with Spatial-Contextual Information for Local Climate Zones

Local climate zones (LCZ) have become a generic criterion for climate analysis among global cities, as they can describe not only the urban climate but also the morphology inside the city. LCZ mapping based on the remote sensing classification method is a fundamental task, and the protocol proposed by the World Urban Database and Access Portal Tools (WUDAPT) project, which consists of random forest classification and filter-based spatial smoothing, is the most common approach. However, the classification and spatial smoothing lack a unified framework, which causes the appearance of small, isolated areas in the LCZ maps. In this paper, a spatial-contextual information-based self-training classification framework (SCSF) is proposed to solve this LCZ classification problem. In SCSF, conditional random field (CRF) is used to integrate the classification and spatial smoothing processing into one model and a self-training method is adopted, considering that the lack of sufficient expert-labeled training samples is always a big issue, especially for the complex LCZ scheme. Moreover, in the unary potentials of CRF modeling, pseudo-label selection using a self-training process is used to train the classifier, which fuses the regional spatial information through segmentation and the local neighborhood information through moving windows to provide a more reliable probabilistic classification map. In the pairwise potential function, SCSF can effectively improve the classification accuracy by integrating the spatial-contextual information through CRF. The experimental results prove that the proposed framework is efficient when compared to the traditional mapping product of WUDAPT in LCZ classification.

A self-training method based on density peaks and an extended parameter-free local noise filter for k nearest neighbor

Self-training method is one of the relatively successful methodologies of semi-supervised classification. It can exploit both labeled data and unlabeled data to train a satisfactory supervised classifier. Mislabeling is one of the largest challenges in the self-training method and the most common technique for removing mislabeled samples is the local noise filter. However, existing local noise filters used in self-training methods confront following technical defects: parameter dependence and using only labeled data to remove mislabeled samples. To address these shortcomings, this paper proposes a novel self-training method based on density peaks and an extended parameter-free local noise filter (STDPNF). In STDPNF, the self-training method based on density peaks is redesigned to be more suitable for combination with local noise filters. Moreover, a new local noise filter based on natural neighbors is proposed to filter out mislabeled instances. Compared with existing local noise filters used in self-training methods, the one in STDPNF is parameter-free and can remove mislabeled samples by exploiting the information of both labeled data and unlabeled data. We focus on k nearest neighbor as a base classifier. In experiments, we verify the efficiency of STDPNF in improving the performance of the base classifier of k nearest neighbor and the advantage of STDPNF in having the ability to remove mislabeled instances efficiently even when labeled data are not sufficient.

Granulation-based self-training for the semi-supervised classification of remote-sensing images

Collection of quality-labeled training samples in the area of remote sensing is very difficult, costly, time-consuming, and tedious due to various constraints. Classification of remote-sensing images is a challenging task due to the limited availability of quality-labeled samples for the training process. To solve the problem of labeled samples, various semi-supervised techniques have been designed and explored for the classification of remote-sensing images. Self-training is a popular semi-supervised method widely used for the training of supervised classifier with limited labeled and a large pool of unlabeled samples. However, the traditional self-training approach gives poor performance for the classification of remote-sensing images. The traditional self-training method selects samples only on the basis of maximum classification probability criterion which may not improve the classifier accuracy. The effectiveness of the classifiers trained in the self-training fashion depends on the selection of correct, diverse, and informative samples for the labeled training set. In this paper, granular computing concepts have been utilized to improve the self-training approach for the classification of the remote-sensing images. The proposed approach first groups the unlabeled samples into several numbers of granules. After that, a supervised classifier is trained with few labeled samples and the trained classifier is used to select the most confident granules set. The selected most confident granules help to add qualitative samples into the labeled set for the effective training of the classifiers. The experimental results with three benchmark remote-sensing data sets show that the proposed method has produced improvement in the classification accuracy.

Semi-Supervised Self-Training Method Based on an Optimum-Path Forest

Semi-supervised self-training method can train an effective classifier by exploiting labeled and unlabeled samples. Recently, a self-training method based on density peaks of data (STDP) is proposed. However, it still suffers from some shortcomings to be addressed. For example, STDP is affected by cut-off distance $d_{c}$ . As a result, it is tricky for STDP to select an optimal parameter $d_{c}$ on each data set. Furthermore, STDP has a poor performance on data sets with some variations in density because of cut-off distance $d_{c}$ . In order to solve these problems, we present a new self-training method which connects unlabeled and labeled samples as vertexes of an optimum path forest to discover the underlying structure of feature space. Furthermore, the underlying structure of the feature space is used to guide the self-training method to train a classifier. Compared with STDP, our algorithm is free of parameters and can work better on data sets with some variations in density. Moreover, we are surprised to find that our algorithm also has some advantages in dealing with overlapping data sets. The experimental results on real data sets clearly demonstrate that our algorithm has better performance than some previous works in improving the performance of base classifiers of k-nearest neighbor, support vector machine and cart.

ChangeMyView Through Concessions: Do Concessions Increase Persuasion?

In Discourse Studies concessions are considered among those argumentative strategies that increase persuasion. We aim to empirically test this hypothesis by calculating the distribution of argumentative concessions in persuasive vs. non-persuasive comments from the the ChangeMyView subreddit. This constitutes a challenging task since concessions do not always bear an argumentative role and are expressed through polysemous lexical markers. Drawing from a theoretically-informed typology of concessions, we first conduct a crowdsourcing task to label a set of polysemous lexical markers as introducing an argumentative concession relation or not. Second, we present a self-training method to automatically identify argumentative concessions using linguistically motivated features. While we achieve a moderate F1 of 57.4% via the self-training method, our subsequent error analysis highlights that the self training method is able to generalize and identify other types of concessions that are argumentative, but were not considered in the annotation guidelines. Our findings from the manual labeling and the classification experiments indicate that the type of argumentative concessions we investigated is almost equally likely to be used in winning and losing arguments. While this result seems to contradict theoretical assumptions, we provide some reasons related to the ChangeMyView subreddit.

Multilingual Neural Machine Translation for Low-Resource Languages

In recent years, Neural Machine Translation (NMT) has been shown to be more effective than phrase-based statistical methods, thus quickly becoming the state of the art in machine translation (MT). However, NMT systems are limited in translating low-resourced languages, due to the significant amount of parallel data that is required to learn useful mappings between languages. In this work, we show how the so-called multilingual NMT can help to tackle the challenges associated with low-resourced language translation. The underlying principle of multilingual NMT is to force the creation of hidden representations of words in a shared semantic space across multiple languages, thus enabling a positive parameter transfer across languages. Along this direction, we present multilingual translation experiments with three languages (English, Italian, Romanian) covering six translation directions, utilizing both recurrent neural networks and transformer (or self-attentive) neural networks. We then focus on the zero-shot translation problem, that is how to leverage multi-lingual data in order to learn translation directions that are not covered by the available training material. To this aim, we introduce our recently proposed iterative self-training method, which incrementally improves a multilingual NMT on a zero-shot direction by just relying on monolingual data. Our results on TED talks data show that multilingual NMT outperforms conventional bilingual NMT, that the transformer NMT outperforms recurrent NMT, and that zero-shot NMT outperforms conventional pivoting methods and even matches the performance of a fully-trained bilingual system.

Identifying genotype-phenotype relationships in biomedical text

BackgroundOne important type of information contained in biomedical research literature is the newly discovered relationships between phenotypes and genotypes. Because of the large quantity of literature, a reliable automatic system to identify this information for future curation is essential. Such a system provides important and up to date data for database construction and updating, and even text summarization. In this paper we present a machine learning method to identify these genotype-phenotype relationships. No large human-annotated corpus of genotype-phenotype relationships currently exists. So, a semi-automatic approach has been used to annotate a small labelled training set and a self-training method is proposed to annotate more sentences and enlarge the training set.ResultsThe resulting machine-learned model was evaluated using a separate test set annotated by an expert. The results show that using only the small training set in a supervised learning method achieves good results (precision: 76.47, recall: 77.61, F-measure: 77.03) which are improved by applying a self-training method (precision: 77.70, recall: 77.84, F-measure: 77.77).ConclusionsRelationships between genotypes and phenotypes is biomedical information pivotal to the understanding of a patient’s situation. Our proposed method is the first attempt to make a specialized system to identify genotype-phenotype relationships in biomedical literature. We achieve good results using a small training set. To improve the results other linguistic contexts need to be explored and an appropriately enlarged training set is required.

2210 PMセルフトレーニング(一般セッション)

2210 PMセルフトレーニング(一般セッション)

A New Sparse Representation Framework for Reconstruction of an Isotropic High Spatial Resolution MR Volume From Orthogonal Anisotropic Resolution Scans.

In magnetic resonance (MR), hardware limitations, scan time constraints, and patient movement often result in the acquisition of anisotropic 3-D MR images with limited spatial resolution in the out-of-plane views. Our goal is to construct an isotropic high-resolution (HR) 3-D MR image through upsampling and fusion of orthogonal anisotropic input scans. We propose a multiframe super-resolution (SR) reconstruction technique based on sparse representation of MR images. Our proposed algorithm exploits the correspondence between the HR slices and the low-resolution (LR) sections of the orthogonal input scans as well as the self-similarity of each input scan to train pairs of overcomplete dictionaries that are used in a sparse-land local model to upsample the input scans. The upsampled images are then combined using wavelet fusion and error backprojection to reconstruct an image. Features are learned from the data and no extra training set is needed. Qualitative and quantitative analyses were conducted to evaluate the proposed algorithm using simulated and clinical MR scans. Experimental results show that the proposed algorithm achieves promising results in terms of peak signal-to-noise ratio, structural similarity image index, intensity profiles, and visualization of small structures obscured in the LR imaging process due to partial volume effects. Our novel SR algorithm outperforms the nonlocal means (NLM) method using self-similarity, NLM method using self-similarity and image prior, self-training dictionary learning-based SR method, averaging of upsampled scans, and the wavelet fusion method. Our SR algorithm can reduce through-plane partial volume artifact by combining multiple orthogonal MR scans, and thus can potentially improve medical image analysis, research, and clinical diagnosis.

OA19.03 Identify Lung Adenocarcinoma in Situ among Pulmonary Micro-Nodules through Blood Gene Expression Profiles

The national lung cancer screening test (NLST) confirmed: low dose CT screening could reduce lung cancer mortality. However, the high false-positive rates of LDCT screening, especially the difficulty of diagnosis of micro-nodules with size less than 10 mm highlight the need of complementary biomarkers to discriminate micro-nodular lung cancer from benign pulmonary diseases. The blood gene expression profiles of 46 lung cancer patients, 38 pulmonary lesions and 51 healthy were investigated to identify the lung cancer-specific genetic signatures. The lung cancer patients containing micro-nodules less than 10 mm were surgically and pathologically diagnosed as lung adenocarcinoma in situ. A self-training logistic regression method was used to identify the lung cancer-specific gene signatures as we previously reported. Six genes, including DDX51, PSME2, ACTL6A, GMEB1, FAM200B, GEMIN6, were identified for discriminating lung adenocarcinoma in situ from health and benign pulmonary diseases. The performance of the six-gene panel for diagnosis of lung adenocarcinoma in situ identified was exhibited in Table 1. Through self-training SVM classifier, the logarithmic odds of each sample was calculated and exhibited, in which the cutoff value was set as zero in logarithmic odds for differentiating lung cancer from benign and control group. The predictive model based on 6-gene panel correctly classified 43 of 46 lung cancer, 39 of 42 benign pulmonary diseases with 93% accuracy, 94% sensitivity, and 93% specificity and 0.97 of ROC AUC. The predictive model based on 6-gene panel (DDX51, PSME2, ACTL6A, GMEB1, FAM200B, GEMIN6) can be used for discriminating between the malignant or benign nodules with size less than 10 mm.

Online network traffic classification with incremental learning

Conventional network traffic detection methods based on data mining could not efficiently handle high throughput traffic with concept drift. Data stream mining techniques are able to classify evolving data streams although most techniques require completely labeled data. This paper proposes an improved data stream mining algorithm for online network traffic classification that is able to incrementally learn from both labeled and unlabeled flows. The algorithm uses the concept of incremental k-means and self-training semi-supervised method to continuously update the classification model after receiving new flow instances. The experimental results show that the proposed algorithm is able to classify 325 thousands flow instances per second and achieves up to 91–94 % average accuracy, even when using 10 % of labeled input flows. It is also able to maintain high accuracy even in the presence of concept drifts. Although there are drifts detected in the datasets evaluated using the Drift Detection Method, our proposed method with incremental learning is able to achieve up to 91–94 % accuracy compared to 60–69 % without incremental learning.

PhyloPythiaS+: a self-training method for the rapid reconstruction of low-ranking taxonomic bins from metagenomes.

Background. Metagenomics is an approach for characterizing environmental microbial communities in situ, it allows their functional and taxonomic characterization and to recover sequences from uncultured taxa. This is often achieved by a combination of sequence assembly and binning, where sequences are grouped into ‘bins’ representing taxa of the underlying microbial community. Assignment to low-ranking taxonomic bins is an important challenge for binning methods as is scalability to Gb-sized datasets generated with deep sequencing techniques. One of the best available methods for species bins recovery from deep-branching phyla is the expert-trained PhyloPythiaS package, where a human expert decides on the taxa to incorporate in the model and identifies ‘training’ sequences based on marker genes directly from the sample. Due to the manual effort involved, this approach does not scale to multiple metagenome samples and requires substantial expertise, which researchers who are new to the area do not have.Results. We have developed PhyloPythiaS+, a successor to our PhyloPythia(S) software. The new (+) component performs the work previously done by the human expert. PhyloPythiaS+ also includes a new k-mer counting algorithm, which accelerated the simultaneous counting of 4–6-mers used for taxonomic binning 100-fold and reduced the overall execution time of the software by a factor of three. Our software allows to analyze Gb-sized metagenomes with inexpensive hardware, and to recover species or genera-level bins with low error rates in a fully automated fashion. PhyloPythiaS+ was compared to MEGAN, taxator-tk, Kraken and the generic PhyloPythiaS model. The results showed that PhyloPythiaS+ performs especially well for samples originating from novel environments in comparison to the other methods.Availability. PhyloPythiaS+ in a virtual machine is available for installation under Windows, Unix systems or OS X on: https://github.com/algbioi/ppsp/wiki.

On the stopping criteria for k-Nearest Neighbor in positive unlabeled time series classification problems

Positive unlabeled time series classification has become an important area during the last decade, as often vast amounts of unlabeled time series data are available but obtaining the corresponding labels is difficult. In this situation, positive unlabeled learning is a suitable option to mitigate the lack of labeled examples. In particular, self-training is a widely used technique due to its simplicity and adaptability. Within this technique, the stopping criterion, i.e., the decision of when to stop labeling, is a critical part, especially in the positive unlabeled context. We propose a self-training method that follows the positive unlabeled approach for time series classification and a family of parameter-free stopping criteria for this method. Our proposal uses a graphical analysis, applied to the minimum distances obtained by the k-Nearest Neighbor as the base learner, to estimate the class boundary. The proposed method is evaluated in an experimental study involving various time series classification datasets. The results show that our method outperforms the transductive results obtained by previous models.

Autobiography as Educational Scientific Tool in Penitentiary

In recent times, educators, social workers and researchers are making increasing use of life histories method as significant evidence of life because the autobiography is the most important self-training method. In the context of prison, writing about themselves means to build personal identity of educators. Therefore, the research team analysed adult writing, investigating both the emotionality and the relationships between educator and prisoners. From a methodological point of view, the research focuses on an original autobiographical format and a semi-structured interview, composed by sixteen pre-determined questions. Before the interview, a five frameworks format was delivered in order to make flash backs into the past experiences, with a view to identify events, facts, people etc., within a structured grid used as pattern. Data analyses were based on quantitative and qualitative methods, they were carried out following reviewed and adapted processes laying their rows on grounded theory. Through the various life stories and personal experiences narrated by educators we could discover that the main motivations of this profession lie primarily in the willingness to disseminate the knowledge. Willingness, determination and formative relationship between educators and prisoners are valuable tools to accompany educators in this difficult task. Relationships between educators and prisoners to achieve educational goals, are essentially based on dialogue and autobiographical skills was proved to be very useful. Telling personal stories means taking a moment for themselves, to recharge and give free rein to the emotions and to be shared with others.