Unlabeled Sentences Research Articles

Uncertainty-Aware Contrastive Learning for semi-supervised named entity recognition

Named entity recognition (NER) based on deep neural networks has shown competitive performance when trained on large-scale human-annotated data. However, they face challenges in low-resource settings, where the available labeled data are scarce. A typical solution is pseudo-labeling which assigns pseudo-labels to the certain (i.e., high confidence) tokens of unlabeled sentences while discards the uncertain (i.e., low confidence) ones. But there still have two potential challenges: (1) discarding the uncertain tokens leads to low utilization of unlabeled data; (2) the intrinsic quality-quantity trade-off issue of pseudo-labeling with confidence threshold. In this work, we propose an innovative method named Uncertainty-Aware Contrastive Learning (UACL) for semi-supervised named entity recognition. Specifically, UACL first utilizes a Gaussian-based class-wise token separation mechanism to dynamically distinguish certain and uncertain tokens, which can self-adaptively adjust the confidence threshold to balance the quantity and quality of pseudo-labeled certain tokens. Then we perform pseudo-supervised learning based on certain tokens and contrastive learning based on uncertain ones, which not only improves the utilization of unlabeled data, but also provides uncertainty-aware guidance information for model training. Furthermore, our method leverages uncertain tokens to optimize token representation, leading to improving performance. The extensive experimental results on four benchmarks demonstrate that the performance of our proposed approach surpasses that of previously leading low-resource baselines.

Use of Natural Language Processing to Identify Sexual and Reproductive Health Information in Clinical Text.

This study aimed to enable clinical researchers without expertise in natural language processing (NLP) to extract and analyze information about sexual and reproductive health (SRH), or other sensitive health topics, from large sets of clinical notes. (1) We retrieved text from the electronic health record as individual notes. (2) We segmented notes into sentences using one of scispaCy's NLP toolkits. (3) We exported sentences to the labeling application Watchful and annotated subsets of these as relevant or irrelevant to various SRH categories by applying a combination of regular expressions and manual annotation. (4) The labeled sentences served as training data to create machine learning models for classifying text; specifically, we used spaCy's default text classification ensemble, comprising a bag-of-words model and a neural network with attention. (5) We applied each model to unlabeled sentences to identify additional references to SRH with novel relevant vocabulary. We used this information and repeated steps 3 to 5 iteratively until the models identified no new relevant sentences for each topic. Finally, we aggregated the labeled data for analysis. This methodology was applied to 3,663 Child Neurology notes for 971 female patients. Our search focused on six SRH categories. We validated the approach using two subject matter experts, who independently labeled a sample of 400 sentences. Cohen's kappa values were calculated for each category between the reviewers (menstruation: 1, sexual activity: 0.9499, contraception: 0.9887, folic acid: 1, teratogens: 0.8864, pregnancy: 0.9499). After removing the sentences on which reviewers did not agree, we compared the reviewers' labels to those produced via our methodology, again using Cohen's kappa (menstruation: 1, sexual activity: 1, contraception: 0.9885, folic acid: 1, teratogens: 0.9841, pregnancy: 0.9871). Our methodology is reproducible, enables analysis of large amounts of text, and has produced results that are highly comparable to subject matter expert manual review.

Weakly Supervised Learning Approach for Implicit Aspect Extraction

Aspect-based sentiment analysis (ABSA) is a process to extract an aspect of a product from a customer review and identify its polarity. Most previous studies of ABSA focused on explicit aspects, but implicit aspects have not yet been the subject of much attention. This paper proposes a novel weakly supervised method for implicit aspect extraction, which is a task to classify a sentence into a pre-defined implicit aspect category. A dataset labeled with implicit aspects is automatically constructed from unlabeled sentences as follows. First, explicit sentences are obtained by extracting explicit aspects from unlabeled sentences, while sentences that do not contain explicit aspects are preserved as candidates of implicit sentences. Second, clustering is performed to merge the explicit and implicit sentences that share the same aspect. Third, the aspect of the explicit sentence is assigned to the implicit sentences in the same cluster as the implicit aspect label. Then, the BERT model is fine-tuned for implicit aspect extraction using the constructed dataset. The results of the experiments show that our method achieves 82% and 84% accuracy for mobile phone and PC reviews, respectively, which are 20 and 21 percentage points higher than the baseline.

S[formula omitted]map: Semisupervised aspect-based sentiment analysis with masked aspect prediction

Aspect-based sentiment analysis (ABSA) refers to a fine-grained task of detecting the sentiment polarities of sentences at the aspect level. To resolve this task, ABSA training samples must be annotated with aspect words and the corresponding sentiment polarities. However, collecting such fine-grained training samples is expensive and time-consuming. Therefore the available ABSA training samples are often scarce. To break the data scarcity challenge of ABSA, we investigate semi-supervised aspect-based sentiment analysis (SemiABSA), which trains ABSA models using a limited amount of expensive labeled sentences and more unlabeled-yet-cheaper sentences. We propose a novel SemiABSA framework, namely semi-supervised aspect-based sentiment analysis with masked aspect prediction (S3map), built on the self-training paradigm. We form pseudo-aspect words and pseudo-sentiment polarities for unlabeled sentences and improve model training. Specifically, a BERT-encoder-based masked aspect prediction (MAP) task achieves the pseudo-aspect words generation. Based on S3map, we thoroughly investigate the potential of SemiABSA from various perspectives. The empirical results show that S3map can consistently improve performance by leveraging unlabeled sentences, even those from different domains.

TaxonPrompt: Taxonomy-aware curriculum prompt learning for few-shot event classification

Event classification (EC) aims to assign the event labels to unlabeled sentences and tends to struggle in real-world applications when only a few annotated samples are available. Previous studies have mainly focused on using meta-learning to overcome the low-resource problem where label data from other tasks are still required for model learning and selection. Accordingly, prompt learning-based approaches are proposed to address the low-resource issue. However, such approaches generally ignore task-specific information and adopt demonstration learning for fine-tuning, which fails to leverage the most informative examples for training and hurts performance. Thus, we propose a taxonomy-aware prompt learning framework TaxonPrompt that trains the language model with samples from easy to hard by imitating the human curricula, which effectively alleviates the classification bottleneck caused by insufficient data. We first design an event prompt generation (EPG) for automatically generating task-specific templates using sentences, labels, and trigger words. Then, we propose a Fisher information-based demonstration filtering (FDF) to dynamically select the most informative support examples for each query to train the model. We have conducted extensive experiments on two EC datasets: FewEvent and RAMS. The experimental results demonstrate the superiority of the proposed model over state-of-the-art baselines. In particular, our approach works well in the scenario of an extremely small number of available task resources and therefore constitutes a solution for few-shot event classification.

AugPrompt: Knowledgeable augmented-trigger prompt for few-shot event classification

Few-Shot Event Classification (FSEC) aims at assigning event labels to unlabeled sentences when limited annotated samples are available. Existing works mainly focus on using meta-learning to overcome the low-resource problem that still requires abundant held-out classes for model learning and selection. Thus we propose to deal with the low-resource problem by utilizing prompts. Further, existing methods suffer from severe trigger biases that may result in ignorance of the context. That is, the correct classifications are gained by looking at only the triggers, which hurts the model’s generalization ability. Thus, we propose a knowledgeable augmented-trigger prompt FSEC framework (AugPrompt), which can overcome the bias issues and alleviates the classification bottleneck brought by insufficient data. In detail, we first design an External Knowledge Injection (EKI) module to incorporate an external knowledge base (Related Words) for trigger augmentation. Then, we propose an Event Prompt Generation (EPG) module to generate appropriate discrete prompts for initializing the continuous prompts. After that, we propose an Event Prompt Tuning (EPT) module to automatically search prompts in the continuous space for FSEC and finally predict the corresponding event types of the inputs. We conduct extensive experiments on two public English datasets for FSEC, i.e., FewEvent and RAMS. The experimental results show the superiority of our proposal over the competitive baselines, where the maximum accuracy increase compared to the strongest baseline reaches 10.8%.

Investigating Prompt Learning for Chinese Few-Shot Text Classification with Pre-Trained Language Models

Text classification aims to assign predefined labels to unlabeled sentences, which tend to struggle in real-world applications when only a few annotated samples are available. Previous works generally focus on using the paradigm of meta-learning to overcome the classification difficulties brought by insufficient data, where a set of auxiliary tasks is given. Accordingly, prompt-based approaches are proposed to deal with the low-resource issue. However, existing prompt-based methods mainly focus on English tasks, which generally apply English pretrained language models that can not directly adapt to Chinese tasks due to structural and grammatical differences. Thus, we propose a prompt-based Chinese text classification framework that uses generated natural language sequences as hints, which can alleviate the classification bottleneck well in low-resource scenarios. In detail, we first design a prompt-based fine-tuning together with a novel pipeline for automating prompt generation in Chinese. Then, we propose a refined strategy for dynamically and selectively incorporating demonstrations into each context. We present a systematic evaluation for analyzing few-shot performance on a wide range of Chinese text classification tasks. Our approach makes few assumptions about task resources and expertise and therefore constitutes a powerful, task-independent approach for few-shot learning.

Self-feeding training method for semi-supervised grammatical error correction

Grammatical error correction (GEC) has been successful with deep and complex neural machine translation models, but the annotated data to train the model are scarce. We propose a novel self-feeding training method that generates incorrect sentences from freely available correct sentences. The proposed training method can generate appropriate wrong sentences from unlabeled sentences, using a data generation model trained as an autoencoder. It can also add artificial noise to correct sentences to automatically generate incorrect sentences. We show that the GEC models trained with the self-feeding training method are successful without extra annotated data or deeper neural network-based models, achieving F0.5 score of 0.5982 on the CoNLL-2014 Shared Task test data with a transformer model. The results also show that fully unlabeled training is possible for data-scarce domains and languages.

Dual Learning for Semi-Supervised Natural Language Understanding

Natural language understanding (NLU) converts sentences into structured semantic forms. The paucity of annotated training samples is still a fundamental challenge of NLU. To solve this data sparsity problem, previous work based on semi-supervised learning mainly focuses on exploiting unlabeled sentences. In this work, we introduce a dual task of NLU, semantic-to-sentence generation (SSG), and propose a new framework for semi-supervised NLU with the corresponding dual model. The framework is composed of dual pseudo-labeling and dual learning method, which enables an NLU model to make full use of data (labeled and unlabeled) through a closed-loop of the primal and dual tasks. By incorporating the dual task, the framework can exploit pure semantic forms as well as unlabeled sentences, and further improve the NLU and SSG models iteratively in the closed-loop. The proposed approaches are evaluated on two public datasets (ATIS and SNIPS). Experiments in the semi-supervised setting show that our methods can outperform various baselines significantly, and extensive ablation studies are conducted to verify the effectiveness of our framework. Finally, our method can also achieve the state-of-the-art performance on the two datasets in the supervised setting. Our code is available at \url{https://github.com/rhythmcao/slu-dual-learning.git}.

Learning an Executable Neural Semantic Parser

This article describes a neural semantic parser that maps natural language utterances onto logical forms that can be executed against a task-specific environment, such as a knowledge base or a database, to produce a response. The parser generates tree-structured logical forms with a transition-based approach, combining a generic tree-generation algorithm with domain-general grammar defined by the logical language. The generation process is modeled by structured recurrent neural networks, which provide a rich encoding of the sentential context and generation history for making predictions. To tackle mismatches between natural language and logical form tokens, various attention mechanisms are explored. Finally, we consider different training settings for the neural semantic parser, including fully supervised training where annotated logical forms are given, weakly supervised training where denotations are provided, and distant supervision where only unlabeled sentences and a knowledge base are available. Experiments across a wide range of data sets demonstrate the effectiveness of our parser.

Improving Neural Chinese Word Segmentation Using Unlabeled Data

Supervised word segmentation heavily relies on large-scale and high quality labeled data. However, building such a corpus is difficult, especially with respect to domain specific data. In this paper, we propose a novel semi-supervised Chinese word segmentation (CWS) method. Specifically, we seek to select more useful sample sentences from the large-scale unlabeled sentences to extend the training data, by means of a sampling strategy that uses character-based semantic similarity. The presented similarity algorithm is used to calculate the similarity between unlabeled sentences and the training data, which can help select helpful sample sentences from unlabeled data. In addition, we integrate an attention mechanism into our word segmentation model to focus on available contextual information. Experiments on PKU, MSR and Weibo benchmark data sets show that our method outperforms the previous neural network models and state-of-the-art methods.

A robust data-driven approach for gene ontology annotation.

Gene ontology (GO) and GO annotation are important resources for biological information management and knowledge discovery, but the speed of manual annotation became a major bottleneck of database curation. BioCreative IV GO annotation task aims to evaluate the performance of system that automatically assigns GO terms to genes based on the narrative sentences in biomedical literature. This article presents our work in this task as well as the experimental results after the competition. For the evidence sentence extraction subtask, we built a binary classifier to identify evidence sentences using reference distance estimator (RDE), a recently proposed semi-supervised learning method that learns new features from around 10 million unlabeled sentences, achieving an F1 of 19.3% in exact match and 32.5% in relaxed match. In the post-submission experiment, we obtained 22.1% and 35.7% F1 performance by incorporating bigram features in RDE learning. In both development and test sets, RDE-based method achieved over 20% relative improvement on F1 and AUC performance against classical supervised learning methods, e.g. support vector machine and logistic regression. For the GO term prediction subtask, we developed an information retrieval-based method to retrieve the GO term most relevant to each evidence sentence using a ranking function that combined cosine similarity and the frequency of GO terms in documents, and a filtering method based on high-level GO classes. The best performance of our submitted runs was 7.8% F1 and 22.2% hierarchy F1. We found that the incorporation of frequency information and hierarchy filtering substantially improved the performance. In the post-submission evaluation, we obtained a 10.6% F1 using a simpler setting. Overall, the experimental analysis showed our approaches were robust in both the two tasks.

Representation for dialect recognition using topographic independent component analysis

In dialect speech recognition, the feature of tone in one dialect is subject to changes in pitch frequency as well as the length of tone. It is beneficial for the recognition if a representation can be derived to account for the frequency and length changes of tone in an effective and meaningful way. In this paper, we propose a method for learning such a representation from a set of unlabeled speech sentences containing the features of the dialect changed from various pitch frequencies and time length. Topographic independent component analysis (TICA) is applied for the unsupervised learning to produce an emergent result that is a topographic matrix made up of basis components. The dialect speech is topographic in the following sense: the basis components as the units of the speech are ordered in the feature matrix such that components of one dialect are grouped in one axis and changes in time windows are accounted for in the other axis. This provides a meaningful set of basis vectors that may be used to construct dialect subspaces for dialect speech recognition.