Semi-supervised Text Classification Research Articles

Improving Semi-Supervised Text Classification with Dual Meta-Learning

The goal of semi-supervised text classification (SSTC) is to train a model by exploring both a small number of labeled data and a large number of unlabeled data, such that the learned semi-supervised classifier performs better than the supervised classifier trained on solely the labeled samples. Pseudo-labeling is one of the most widely used SSTC techniques, which trains a teacher classifier with a small number of labeled examples to predict pseudo labels for the unlabeled data. The generated pseudo-labeled examples are then utilized to train a student classifier, such that the learned student classifier can outperform the teacher classifier. Nevertheless, the predicted pseudo labels may be inaccurate, making the performance of the student classifier degraded. The student classifier may perform even worse than the teacher classifier. To alleviate this issue, in this paper, we introduce a dual meta-learning ( DML ) technique for semi-supervised text classification, which improves the teacher and student classifiers simultaneously in an iterative manner. Specifically, we propose a meta-noise correction method to improve the student classifier by proposing a Noise Transition Matrix (NTM) with meta-learning to rectify the noisy pseudo labels. In addition, we devise a meta pseudo supervision method to improve the teacher classifier. Concretely, we exploit the feedback performance from the student classifier to further guide the teacher classifier to produce more accurate pseudo labels for the unlabeled data. In this way, both teacher and student classifiers can co-evolve in the iterative training process. Extensive experiments on four benchmark datasets highlight the effectiveness of our DML method against existing state-of-the-art methods for semi-supervised text classification. We release our code and data of this paper publicly at https://github.com/GRIT621/DML.

Towards Robust Learning with Noisy and Pseudo Labels for Text Classification

Unlike Positive Training (PT), Negative Training (NT) is an indirect learning technique that trains the model on a combination of clean and noisy data using complementary labels, which are randomly generated from the label space except for the actual label. Although clean samples have identical distributions to the test samples, they are treated with the same level of uncertainty as noisy samples because of the complementary labeling of NT. Consequently, their contribution to the overall performance is relatively lower. We propose a Learning with Noisy and Pseudo Label (LNPL) framework, which jointly trains the model using PT and NT on clean and noisy data, respectively. We aim to enable direct learning on clean samples while leveraging the robustness of NT against noise in a unified framework. To mitigate the abundance of noisy instances, we leverage a gradient reversal layer at the top of LNPL as a regularization term to mislead the recognition of the source of the instance (e.g., clean or noisy). Moreover, we introduce a self-training LNPL that performs a semi-supervised text classification task as a learning with noisy pseudo-label problem. Extensive experiments on various textual benchmark datasets demonstrate that LNPL is robust and consistently outperforms the alternatives. The code is available on GitHub.1

CDGAN-BERT: Adversarial constraint and diversity discriminator for semi-supervised text classification

Semi-supervised generative adversarial learning (SS-GAN) significantly improves the model’s performance with limited labeled data, especially on text classification. However, these SS-GAN-like works pay more attention to the more real properties of the generated samples and ignore their adversarial nature, which may prevent the model from learning more higher-quality data representations. And the existing single discriminator may produce discrimination ambiguity when discriminating against tasks with different discriminative properties (simultaneously identifying real or fake and classifying categories), which may harm the model’s classification performance. In this paper, we propose a novel CDGAN-BERT, an SS-GAN-based architecture with the adversarial constraint and a diversity discriminator. Specifically, CDGAN-BERT first focuses on adversarial constraint, which calculates the model’s intrinsic state representation with the reciprocal of Mean Squared Error (MSE), further keeping adversarial of the generated samples relative to the real data distribution. Then, a diversity discriminator is designed to improve the model’s classification performance by alleviating discriminative ambiguity, which refines tasks with different discriminative attributes by adding an additional authenticity discriminator. We validate our model on 6 text datasets and achieve significant improvements, especially with limited supervision. The experimental results show that our method outperforms or gets comparable results to other state-of-the-art semi-supervised learning methods on several datasets. Especially for the datasets of QC-Fine and QC-Coarse with limited labeled data, our CDGAN-BERT achieves the best average Micro-F1 scores of 72.5% and 92.037%, respectively.

Rank-Aware Negative Training for Semi-Supervised Text Classification

Abstract Semi-supervised text classification-based paradigms (SSTC) typically employ the spirit of self-training. The key idea is to train a deep classifier on limited labeled texts and then iteratively predict the unlabeled texts as their pseudo-labels for further training. However, the performance is largely affected by the accuracy of pseudo-labels, which may not be significant in real-world scenarios. This paper presents a Rank-aware Negative Training (RNT) framework to address SSTC in learning with noisy label settings. To alleviate the noisy information, we adapt a reasoning with uncertainty-based approach to rank the unlabeled texts based on the evidential support received from the labeled texts. Moreover, we propose the use of negative training to train RNT based on the concept that “the input instance does not belong to the complementary label”. A complementary label is randomly selected from all labels except the label on-target. Intuitively, the probability of a true label serving as a complementary label is low and thus provides less noisy information during the training, resulting in better performance on the test data. Finally, we evaluate the proposed solution on various text classification benchmark datasets. Our extensive experiments show that it consistently overcomes the state-of-the-art alternatives in most scenarios and achieves competitive performance in the others. The code of RNT is publicly available on GitHub.

Commonsense knowledge powered heterogeneous graph attention networks for semi-supervised short text classification

In real-world scenarios, considerable human power and expert knowledge are required to label data. Therefore, solving short text classification problems in a semi-supervised manner is a good method. Existing graph-based semi-supervised short text classification models can achieve good classification performance. However, these models do not introduce some prior knowledge related to short texts, such as commonsense knowledge and concepts, from the knowledge bases to alleviate the data sparsity of short texts, which limits the improvement of classification performance. To overcome the above limitations, this paper addresses the problem of semi-supervised short text classification by proposing a novel model called the Commonsense Knowledge-Powered Heterogeneous Graph Attention Network (CSK-HGAT). This model mines the short text-related commonsense knowledge from the definitions corresponding to extracted short text keywords and obtains the short text conceptual information from the existing knowledge bases by conceptualizing the identified short text entities. Next, the embeddings are generated separately for short texts, commonsense knowledge, and concepts. Then, a heterogeneous information network (HIN) containing diverse knowledge is constructed by using the semantic relations among the short texts, commonsense knowledge and the concepts. Finally, this HIN is embedded into the heterogeneous graph attention network model (containing node-level and type-level attentions) to fully utilize the mined prior knowledge (i.e., commonsense knowledge and concepts) to achieve effective semi-supervised short text classification. Experimental results on four benchmark datasets show that the proposed CSK-HGAT model outperforms state-of-the-art semi-supervised short text classification models in terms of classification accuracy.

An embedding-based text classification approach for understanding micro-geographic housing dynamics

In this article, we introduce an approach for studying micro-geographic housing dynamics using an embedding-based, semi-supervised text classification approach on longitudinal, point-level property listing data. Based on the text used to describe properties for sale and a set of predefined classes and keywords, listings are classified according to their lifecycle of investment or disinvestment. The mixture of property types within 1 × 1 mile grid cells are then calculated and used as input in a clustering algorithm to develop a place-based classification that enables us to examine patterns of change over time. In a case study on Mecklenburg County, North Carolina using 158,253 real estate listings between 2001 and 2020, we demonstrate how this approach has the potential to further our understanding of housing and neighborhood dynamics by grounding our analysis in theoretical concepts around the housing lifecycle and its relationship to neighborhood change.

Multi-MCCR: Multiple models regularization for semi-supervised text classification with few labels

Semi-supervised learning has achieved impressive results and is commonly applied in text classifications. However, in situations where labeled texts are exceedingly limited, neural networks are prone to over-fitting due to the non-negligible inconsistency between model training and inference caused by dropout mechanisms that randomly mask some neurons. To alleviate this inconsistency problem, we propose a simple Multiple Models Contrast learning based on Consistent Regularization, named Multi-MCCR, which consists of multiple models with the same structure and a C-BiKL loss strategy. Specifically, one sample first goes through multiple identical models to obtain multiple different output distributions, which enriches the sample output distributions and provides conditions for subsequent consistency approximation. Then, the C-BiKL loss strategy is proposed to minimize the combination of the bidirectional Kullback−−Leibler (BiKL) divergence between the above multiple output distributions and the Cross-Entropy loss on labeled data, which provides consistency constraints (BiKL) for the model and simultaneously ensures correct classification (Cross-Entropy). Through the above setting of multi-model contrast learning, the inconsistency caused by the randomness of dropout between model training and inference is alleviated, thereby avoiding over-fitting and improving the classification ability in scenarios with limited labeled samples. We conducted experiments on six widely-used text classification datasets, including sentiment analysis, topic categorization, and reviews classification, and the experimental results show that our method is universally effective in semi-supervised text classification with limited labeled texts.

A multi-semantic passing framework for semi-supervised long text classification

A multi-semantic passing framework for semi-supervised long text classification

Accelerating Semi-Supervised Text Classification by K-Way Projecting Networks

The state of the art semi-supervised learning framework has greatly shown its potential in making deep and complex language models such as BERT highly effective for text classification tasks when labeled data is limited. However, the large size and low inference speed of such models may hinder their application on resources-limited or real-time use cases. In this paper, we propose a new approach in semi-supervised learning framework to distill large complex teacher model into a fairly lightweight student model which has the ability of acquiring knowledge from different layers of teacher with the usage of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> -way projecting networks. Across four English datasets in text classification benchmarks and one dataset collected from an Chinese online course, our experiment shows that this student model achieves comparable results with the state of the art Transformer-based semi-supervised text classification methods, while using only 0.156MB parameters and having an inference speed 785 times faster than the teacher model.

LMGAN: Linguistically Informed Semi-Supervised GAN with Multiple Generators.

Semi-supervised learning is one of the active research topics these days. There is a trial that solves semi-supervised text classification with a generative adversarial network (GAN). However, its generator has a limitation in producing fake data distributions that are similar to real data distributions. Since the real data distribution is frequently changing, the generator could not create adequate fake data. To overcome this problem, we present a novel approach for semi-supervised learning for text classification based on generative adversarial networks, Linguistically Informed SeMi-Supervised GAN with Multiple Generators, LMGAN. LMGAN uses trained bidirectional encoder representations from transformers (BERT) and the discriminator from GAN-BERT. In addition, LMGAN has multiple generators and utilizes the hidden layers of BERT. To reduce the discrepancy between the distribution of fake data and real data distribution, LMGAN uses fine-tuned BERT and the discriminator from GAN-BERT. However, since injecting fine-tuned BERT could induce incorrect fake data distribution, we utilize linguistically meaningful intermediate hidden layer outputs of BERT to enrich fake data distribution. Our model shows well-distributed fake data compared to the earlier GAN-based approach that failed to generate adequate high-quality fake data. Moreover, we can get better performances with extremely limited amounts of labeled data, up to 20.0%, compared to the baseline GAN-based model.

Ptr4BERT: Automatic Semisupervised Chinese Government Message Text Classification Method Based on Transformer-Based Pointer Generator Network

With the development of Internet technology, government affairs can be handled online. More and more citizens are using online platforms to report to government departments, which is generating a lot of textual data. Among them, the basic but important problem is to automatically classify the different categories of messages, so that staff from different departments can process relevant information quickly. However, government messages have problems such as fast update rate, a large amount of information, long texts, and difficulty in capturing key points, which make supervised learning methods unsuitable for processing such texts. To address these problems, we propose a semisupervised text classification method based on a transformer-based pointer generator network named Ptr4BERT, which uses the pointer generator network with BERT(bidirectional encoder representation from transformers) embedding as a preprocessor for feature extraction. In this method, text classification can achieve very good results with a small set of labeled data, by extracting features exclusively from the message text. In order to verify the effect of our proposed model, we performed some experiments. Besides, we designed a crawler program and obtained two datasets from different websites, which are named HNMes and QDMes. Experimental results have shown that the proposed method outperforms the state-of-the-art methods significantly.

Self-training method based on GCN for semi-supervised short text classification

Semi-supervised short text classification is a challenging problem due to the sparsity and limited labeled data. Due to the lack of labeled data, many models focus on the generation of text samples, which is cumbersome and has poor scalability. To overcome this deficiency, in this paper, we propose a Self-Training Text method based on Graph Convolutional Networks (ST-Text-GCN). Differently from the previous literature, our self-training method is convenient. The labeled information is propagated to target samples along the structure of the manifold, instead of introducing the extra knowledge. Specifically, instead of adding text training samples, our method adds keywords to training set. The model will calculate the confidence of each word. Confidence indicates the degree of ambiguity of a word. Some words with high confidence are automatically marked as pseudo-labeled data. Meanwhile, word confidence is added to the calculation of the edge weights of the graph to reduce the classification error caused by word ambiguity. Our method makes full use of the keywords in short texts when labeled data is scarce. Extensive experimental results have demonstrated that our proposed method outperforms state-of-the-art models on multiple benchmark datasets.

Contrast-Enhanced Semi-supervised Text Classification with Few Labels

Traditional text classification requires thousands of annotated data or an additional Neural Machine Translation (NMT) system, which are expensive to obtain in real applications. This paper presents a Contrast-Enhanced Semi-supervised Text Classification (CEST) framework under label-limited settings without incorporating any NMT systems. We propose a certainty-driven sample selection method and a contrast-enhanced similarity graph to utilize data more efficiently in self-training, alleviating the annotation-starving problem. The graph imposes a smoothness constraint on the unlabeled data to improve the coherence and the accuracy of pseudo-labels. Moreover, CEST formulates the training as a “learning from noisy labels” problem and performs the optimization accordingly. A salient feature of this formulation is the explicit suppression of the severe error propagation problem in conventional semi-supervised learning. With solely 30 labeled data per class for both training and validation dataset, CEST outperforms the previous state-of-the-art algorithms by 2.11% accuracy and only falls within the 3.04% accuracy range of fully-supervised pre-training language model fine-tuning on thousands of labeled data.

Generative Adversarial learning with Negative Data Augmentation for Semi-supervised Text Classification

In recent years, semi-supervised generative adversarial networks (SS-GANs) models such as GAN-BERT have achieved promising results on the text classification task. One of the techniques used in these models to mitigate the generator from mode collapse is feature matching (FM). Although FM addresses some of the critical issues of SS-GANs, these models still suffer from mode collapse with missing coverage outside the data manifold. Moreover, FM loosely tries to match the distribution between the real data and the fake generated samples. By doing this, the generator can generate fake samples inside high-density regions in the data manifold, where the discriminator learns to misclassify them as out-of-data-manifold regions. In this work, we employ the negative data augmentation (NDA) technique, for the first time in text classification, to alleviate the mentioned problems. NDA is a unique way of producing out-of-distribution fake examples by applying mixup transformation on the fake samples and augmented real data. In our new model (NDA-GAN), we produce NDA samples by combining the generator's output with the contextual representation of the real data. As a result of the mixing, NDA samples are less likely to place in the high-density regions, and due to blending with real data representations, these samples reasonably preserve a close distance to the data manifold. Consequently, the NDA samples increase the discriminator's power to find the optimal decision boundary. Our experimental results demonstrate that the negative augmented samples improve the overall accuracy of our proposed model and make it more confident when detecting out-of-distribution samples.&#x0D;

Graph Convolutional Network Based on Multi-Head Pooling for Short Text Classification

The short text, sparse features, and the lack of training data, etc. are still the key bottlenecks that restrict the successful application of traditional text classification methods. To address these problems, we propose a Multi-head-Pooling-based Graph Convolutional Network (MP-GCN) for semi-supervised short text classification, and introduce its three architectures, which focus on the node representation learning of 1-order, 1&2-order of isomorphic graphs, and 1-order of heterogeneous graphs, respectively. It only focuses on the structural information of the text graph and does not need pre-training word embedding as the initial node feature. A graph pooling based on self-attention is introduced to evaluate and select important nodes, and the multi-head method is used to provide multiple representation subspaces for pooling without adding trainable parameters. Experimental results demonstrated that, without using pre-training embedding, MP-GCN outperforms state-of-the-art models across five benchmark datasets.

SALNet: Semi-supervised Few-Shot Text Classification with Attention-based Lexicon Construction

We propose a semi-supervised bootstrap learning framework for few-shot text classification. From a small amount of the initial dataset, our framework obtains a larger set of reliable training data by using the attention weights from an LSTM-based trained classifier. We first train an LSTM-based text classifier from a given labeled dataset using the attention mechanism. Then, we collect a set of words for each class called a lexicon, which is supposed to be a representative set of words for each class based on the attention weights calculated for the classification task. We bootstrap the classifier using the new data that are labeled by the combination of the classifier and the constructed lexicons to improve the prediction accuracy. As a result, our approach outperforms the previous state-of-the-art methods including semi-supervised learning algorithms and pretraining algorithms for few-shot text classification task on four publicly available benchmark datasets. Moreover, we empirically confirm that the constructed lexicons are reliable enough and substantially improve the performance of the original classifier.

HGAT: Heterogeneous Graph Attention Networks for Semi-supervised Short Text Classification

Short text classification has been widely explored in news tagging to provide more efficient search strategies and more effective search results for information retrieval. However, most existing studies, concentrating on long text classification, deliver unsatisfactory performance on short texts due to the sparsity issue and the insufficiency of labeled data. In this article, we propose a novel heterogeneous graph neural network-based method for semi-supervised short text classification, leveraging full advantage of limited labeled data and large unlabeled data through information propagation along the graph. Specifically, we first present a flexible heterogeneous information network (HIN) framework for modeling short texts, which can integrate any type of additional information and meanwhile capture their relations to address the semantic sparsity. Then, we propose Heterogeneous Graph Attention networks (HGAT) to embed the HIN for short text classification based on a dual-level attention mechanism, including node-level and type-level attentions. To efficiently classify new coming texts that do not previously exist in the HIN, we extend our model HGAT for inductive learning, avoiding re-training the model on the evolving HIN. Extensive experiments on single-/multi-label classification demonstrates that our proposed model HGAT significantly outperforms state-of-the-art methods across the benchmark datasets under both transductive and inductive learning.

Semi-Supervised Text Classification Framework: An Overview of Dengue Landscape Factors and Satellite Earth Observation.

In recent years there has been an increasing use of satellite Earth observation (EO) data in dengue research, in particular the identification of landscape factors affecting dengue transmission. Summarizing landscape factors and satellite EO data sources, and making the information public are helpful for guiding future research and improving health decision-making. In this case, a review of the literature would appear to be an appropriate tool. However, this is not an easy-to-use tool. The review process mainly includes defining the topic, searching, screening at both title/abstract and full-text levels and data extraction that needs consistent knowledge from experts and is time-consuming and labor intensive. In this context, this study integrates the review process, text scoring, active learning (AL) mechanism, and bidirectional long short-term memory (BiLSTM) networks, and proposes a semi-supervised text classification framework that enables the efficient and accurate selection of the relevant articles. Specifically, text scoring and BiLSTM-based active learning were used to replace the title/abstract screening and full-text screening, respectively, which greatly reduces the human workload. In this study, 101 relevant articles were selected from 4 bibliographic databases, and a catalogue of essential dengue landscape factors was identified and divided into four categories: land use (LU), land cover (LC), topography and continuous land surface features. Moreover, various satellite EO sensors and products used for identifying landscape factors were tabulated. Finally, possible future directions of applying satellite EO data in dengue research in terms of landscape patterns, satellite sensors and deep learning were proposed. The proposed semi-supervised text classification framework was successfully applied in research evidence synthesis that could be easily applied to other topics, particularly in an interdisciplinary context.

Adversarial Dropout for Recurrent Neural Networks

Successful application processing sequential data, such as text and speech, requires an improved generalization performance of recurrent neural networks (RNNs). Dropout techniques for RNNs were introduced to respond to these demands, but we conjecture that the dropout on RNNs could have been improved by adopting the adversarial concept. This paper investigates ways to improve the dropout for RNNs by utilizing intentionally generated dropout masks. Specifically, the guided dropout used in this research is called as adversarial dropout, which adversarially disconnects neurons that are dominantly used to predict correct targets over time. Our analysis showed that our regularizer, which consists of a gap between the original and the reconfigured RNNs, was the upper bound of the gap between the training and the inference phases of the random dropout. We demonstrated that minimizing our regularizer improved the effectiveness of the dropout for RNNs on sequential MNIST tasks, semi-supervised text classification tasks, and language modeling tasks.

Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function

In this paper, we study bidirectional LSTM network for the task of text classification using both supervised and semisupervised approaches. Several prior works have suggested that either complex pretraining schemes using unsupervised methods such as language modeling (Dai and Le 2015; Miyato, Dai, and Goodfellow 2016) or complicated models (Johnson and Zhang 2017) are necessary to achieve a high classification accuracy. However, we develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results compared with more complex approaches. Furthermore, in addition to cross-entropy loss, by using a combination of entropy minimization, adversarial, and virtual adversarial losses for both labeled and unlabeled data, we report state-of-theart results for text classification task on several benchmark datasets. In particular, on the ACL-IMDB sentiment analysis and AG-News topic classification datasets, our method outperforms current approaches by a substantial margin. We also show the generality of the mixed objective function by improving the performance on relation extraction task.1