Sentence Representation Research Articles

A Metric Differential Privacy Mechanism for Sentence Embeddings

Sentence embeddings represent the meaning of a given sentence using a fixed dimensional vector. Different approaches have been proposed in the Natural Language Processing (NLP) community for learning encoders that can produce accurate sentence embeddings that perform well for diverse downstream tasks that require sentence representations. Despite these prior work focusing mainly on creating accurate sentence embeddings, how to protect the privacy of the sensitive information contained in the sentences remains an unexplored research problem. In this paper, we propose, Covering Metric Analytic Gaussian (CMAG), a covering metric Differential Privacy (DP) mechanism for sentence embeddings such that minimal random noise is added to a set of sentence embeddings produced by an encoder to protect the private information expressed in those sentences. Given a sentence embedding s, CMAG considers the Mahalanobis distance between s and the other sentence embeddings s′ in the local neighbourhood of s to determine the minimal amount of random noise that must be added to s to obtain provable metric DP guarantees. Experimental results show that the proposed DP mechanism protects private information better than previously proposed DP mechanisms, while reporting good performance in a broad range of downstream NLP tasks.

Data Poisoning Attack on Black-Box Neural Machine Translation to Truncate Translation.

Neural machine translation (NMT) systems have achieved outstanding performance and have been widely deployed in the real world. However, the undertranslation problem caused by the distribution of high-translation-entropy words in source sentences still exists, and can be aggravated by poisoning attacks. In this paper, we propose a new backdoor attack on NMT models by poisoning a small fraction of parallel training data. Our attack increases the translation entropy of words after injecting a backdoor trigger, making them more easily discarded by NMT. The final translation is part of target translation, and the position of the injected trigger poison affects the scope of the truncation. Moreover, we also propose a defense method, Backdoor Defense by Sematic Representation Change (BDSRC), against our attack. Specifically, we selected backdoor candidates based on the similarity between the semantic representation of words in a sentence and the overall sentence representation. Then, the injected backdoor is identified through computing the semantic deviation caused by backdoor candidates. The experiments show that our attack strategy can achieve a nearly 100% attack success rate, and the functionality of main translation tasks is almost unaffected in models having performance degradation that is less than 1 BLEU. Nonetheless, our defense method can effectively identify backdoor triggers and alleviate performance degradation.

A simple yet effective self-debiasing framework for transformer models

Current Transformer-based natural language understanding (NLU) models heavily rely on dataset biases, while failing to handle real-world out-of-distribution (OOD) instances. Many methods have been proposed to deal with this issue, but they ignore the fact that the features learned in different layers of Transformer-based NLU models are different. In this paper, we first conduct preliminary studies to obtain two conclusions: 1) both low- and high-layer sentence representations encode common biased features during training; 2) the low-layer sentence representations encode fewer unbiased features than the high-layer ones. Based on these conclusions, we propose a simple yet effective self-debiasing framework for Transformer-based NLU models. Concretely, we first stack a classifier on a selected low layer. Then, we introduce a residual connection that feeds the low-layer sentence representation to the top-layer classifier. In this way, the top-layer sentence representation will be trained to ignore the common biased features encoded by the low-layer sentence representation and focus on task-relevant unbiased features. During inference, we remove the residual connection and directly use the top-layer sentence representation to make predictions. Extensive experiments and in-depth analyses on NLU tasks demonstrate the superiority of our framework, achieving a new state-of-the-art (SOTA) on three OOD test sets.

Quantum-inspired semantic matching based on neural networks with the duality of density matrices

Social media text can be semantically matched in different ways, viz paraphrase identification, answer selection, community question answering, and so on. The performance of the above semantic matching tasks depends largely on the ability of language modeling. Neural network based language models and probabilistic language models are two main streams of language modeling approaches. However, few prior work has managed to unify them in a single framework on the premise of preserving probabilistic features during the neural network learning process. Motivated by recent advances of quantum-inspired neural networks for text representation learning, we fill the gap by resorting to density matrices, a key concept describing a quantum state as well as a quantum probability distribution. The state and probability views of density matrices are mapped respectively to the neural and probabilistic aspects of language models. Concretizing this state-probability duality to the semantic matching task, we build a unified neural-probabilistic language model through a quantum-inspired neural network. Specifically, we take the state view to construct a density matrix representation of sentence, and exploit its probabilistic nature by extracting its main semantics, which form the basis of a legitimate quantum measurement. When matching two sentences, each sentence is measured against the main semantics of the other. Such a process is implemented in a neural structure, facilitating an end-to-end learning of parameters. The learned density matrix representation reflects an authentic probability distribution over the semantic space throughout the training process. Experiments show that our model significantly outperforms a wide range of prominent classical and quantum-inspired baselines.

Distantly supervised relation extraction with a Meta-Relation enhanced Contrastive learning framework

Distantly supervised relation extraction employs the alignment of unstructured corpora with knowledge bases to automatically generate labeled data. This method, however, often introduces significant label noise. To address this, multi-instance learning has been widely utilized over the past decade, aiming to extract reliable features from a bag of sentences. Yet, multi-instance learning struggles to effectively distinguish between clean and noisy instances within a bag, thereby hindering the full utilization of informative instances and the reduction of the impact of incorrectly labeled instances. In this paper, we propose a new Meta-Relation enhanced Contrastive learning based method for distantly supervised Relation Extraction named MRConRE. Specifically, we generate a “meta relation pattern” (MRP) for each bag, based on its semantic content, to differentiate between clean and noisy instances. Noisy instances are then transformed into beneficial bag-level instances through relabeling. Subsequently, contrastive learning is employed to develop precise sentence representations, forming the overall representation of the bag. Finally, we utilize a mixup strategy to integrate bag-level information for model training. Our method’s effectiveness is validated through experiments on various benchmarks.

A bi-consolidating model for joint relational triple extraction

Current methods to extract relational triples directly make a prediction based on a possible entity pair in a raw sentence without depending on entity recognition. The task suffers from a serious semantic overlapping problem, in which several relation triples may share one or two entities in a sentence. In this paper, based on a two-dimensional sentence representation, a bi-consolidating model is proposed to address this problem by simultaneously reinforcing the local and global semantic features relevant to a relation triple. This model consists of a local consolidation component and a global consolidation component. The first component uses a pixel difference convolution to enhance semantic information of a possible triple representation from adjacent regions and mitigate noise in neighboring neighbors. The second component strengthens the triple representation based a channel attention and a spatial attention, which has the advantage to learn remote semantic dependencies in a sentence. They are helpful to improve the performance of both entity identification and relation type classification in relation triple extraction. After evaluated on several publish datasets, the bi-consolidating model achieves competitive performance. Analytical experiments demonstrate the effectiveness of our model for relational triple extraction and give motivation for other natural language processing tasks.

Semantic-guided attention and adaptive gating for document-level relation extraction

In natural language processing, document-level relation extraction is a complex task that aims to predict the relationships among entities by capturing contextual interactions from an unstructured document. Existing graph- and transformer-based models capture long-range relational facts across sentences. However, they still cannot fully exploit the semantic information from multiple interactive sentences, resulting in the exclusion of influential sentences for related entities. To address this problem, a novel Semantic-guided Attention and Adaptively Gated (SAAG) model is developed for document-level relation extraction. First, a semantic-guided attention module is designed to guide sentence representation by assigning different attention scores to different words. The multihead attention mechanism is then used to capture the attention of different subspaces further to generate a document context representation. Finally, the SAAG model exploits the semantic information by leveraging a gating mechanism that can dynamically distinguish between local and global contexts. The experimental results demonstrate that the SAAG model outperforms previous models on two public datasets.

Emotion-Semantic-Aware Dual Contrastive Learning for Epistemic Emotion Identification of Learner-Generated Reviews in MOOCs.

Identifying the epistemic emotions of learner-generated reviews in massive open online courses (MOOCs) can help instructors provide adaptive guidance and interventions for learners. The epistemic emotion identification task is a fine-grained identification task that contains multiple categories of emotions arising during the learning process. Previous studies only consider emotional or semantic information within the review texts alone, which leads to insufficient feature representation. In addition, some categories of epistemic emotions are ambiguously distributed in feature space, making them hard to be distinguished. In this article, we present an emotion-semantic-aware dual contrastive learning (ES-DCL) approach to tackle these issues. In order to learn sufficient feature representation, implicit semantic features and human-interpretable emotional features are, respectively, extracted from two different views to form complementary emotional-semantic features. On this basis, by leveraging the experience of domain experts and the input emotional-semantic features, two types of contrastive losses (label contrastive loss and feature contrastive loss) are formulated. They are designed to train the discriminative distribution of emotional-semantic features in the sample space and to solve the anisotropy problem between different categories of epistemic emotions. The proposed ES-DCL is compared with 11 other baseline models on four different disciplinary MOOCs review datasets. Extensive experimental results show that our approach improves the performance of epistemic emotion identification, and significantly outperforms state-of-the-art deep learning-based methods in learning more discriminative sentence representations.

Drug and Vaccine Extractive Text Summarization Insights Using Fine-tuned Transformers

Text representation is a key aspect in determining the success of various text summarizing techniques. Summarization using pre-trained transformer models has produced encouraging results. Yet the scope of applying these models in medical and drug discovery is not examined to a proper extent. To address this issue, this article aims to perform extractive summarization based on fine-tuned transformers pertaining to drug and medical domain. This research also aims to enhance sentence representation. Exploring the extractive text summarization aspects of medical and drug discovery is a challenging task as the datasets are limited. Hence, this research concentrates on the collection of abstracts collected from PubMed for various domains of medical and drug discovery such as drug, COVID, etc.with a total capacity of 1,370 abstracts. A detailed experimentation using BART (Bidirectional Autoregressive Transformer), T5(Text-to-Text Transfer Transformer), LexRank and TexRank for the analysis of the dataset is carried out in this research to perform extractive text summarization.

Extracting Sentence Embeddings from Pretrained Transformer Models

Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT’s hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT’s sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.

HỌC BIỂU DIỄN CÂU SỬ DỤNG MÔ HÌNH LSTM TRONG BÀI TOÁN TÌM KIẾM CÂU HỎI

Learning sentence representation with the full semantics of a document is a challenge in natural language processing problems because if the semantic representation vector of the sentence is suitable, it will increase the performance of finding similar question problems. In this paper, we propose implementing a series of LSTM models with different ways of extracting sentence representations and applying them to question retrieval to exploit the hidden semantics of sentences. These methods give sentence representation from hidden layers of the LSTM model. The results show that the technique using a combination of both Max Pooling and Mean Pooling gives the highest results on the 2017 SemEval dataset for the problem of finding similarity questions. Keywords: LSTM; Deep Learning; NLP; QA; learning sentence representation; CQA.

Cross-biased contrastive learning for answer selection with dual-tower structure

A large number of unanswered products-related questions appear in the E-commerce platforms, necessitating the deployment of question-answering models to automatically provide precise responses for the user. However, the substantial absence of ground truth presents challenges for implementing supervised learning, and existing unsupervised contrastive learning methods have not addressed this issue fundamentally, with limitations in integrating multi-level features. This paper presents a dual-tower cross-biased contrastive learning model for answer selection, named CCLM-AS. By taking questions and augmented product reviews as positive pairs for unsupervised contrastive learning and employing a dual-tower structure encoder, CCLM-AS learns sentence representations from unlabeled data effectively. Furthermore, the designed training objective realizes multi-level feature integration, including character, sentence and dialogue levels, which enhances the model's inference ability. The experimental results on AmazonQA dataset indicate that our proposed model outperforms existing contrastive learning-based sentence representation models and also achieves comparable performance to supervised answer selection models. This demonstrates that CCLM-AS can not only alleviate the data sparsity problem effectively but also retain the excellent performance.

Aspect-based sentiment analysis: A dual-task learning architecture using imbalanced maximized-area under the curve proximate support vector machine and reinforcement learning

Aspect-based sentiment analysis (ABSA) deals with understanding sentiments about specific attributes. Aspect term extraction (ATE) plays an essential role in aspect sentiment classification (ASC) by specifying the main aspects for sentiment accuracy. This paper introduces a dual-task framework for ATE and ASC models in which, the ATE model uses the imbalanced maximized-area under the curve proximate support vector machine (ImAUC-PSVM) method to mitigate the imbalance problem and the output of ATE serves as the reward in the training process of ASC supported by RL (reinforcement learning). The ImAUC-PSVM uses the conventional PSVM (proximate support vector machine) strength for sentiment tasks and the AUC (area under the curve) optimization for imbalance addressing. The ASC model employs multiple kernels for word embedding, with a transductive long short-term memory (TLSTM) for sentence representation that considers samples near the test point for better model refinement. The policy of the RL-based training strategy of the ASC model is represented by a multilayer perceptron (MLP) that categorizes sentences into positive, negative and neutral sentiments. In this RL context, aspect terms serve as rewards in the training process. The scope loss function (SLF) is also integrated within RL to rectify dataset imbalances. To optimize ATE and ASC models, an artificial bee colony (ABC) technique is incorporated for hyperparameter optimization. Experiments and evaluations conducted on English (Restaurant and Laptop) and Hindi datasets show the superiority of the proposed model over existing models of the literature. Moreover, transfer learning (TL) is applied to a Twitter dataset to assess its effect on ATE. The F-measures of the ATE model are obtained as 90.12 and 84.10 and the accuracies of the ASC model are calculated as 84.10 and 82.02 on Restaurant and Laptop datasets, respectively.

Camouflaged named entity recognition in 2D sentence representation

Organizing all spans of a sentence into a two-dimensional (2D) representation unfolds a semantic plane. It has the advantage to resolve nested semantic structures and to build linguistic dependencies across a whole sentence. The main problem is that neighboring elements in the semantic plane are span representations referred to overlapped phrases. Because these representations share the same contextual features in a sentence, a true entity representation is early faded into the background surroundings. It leads to camouflaged named entities in the 2D sentence representation. In this paper, we propose a finer-scale and coarse-scale sentence representation to support camouflaged named entity recognition. The mixed-scale representation has the ability to encode differential clues between entity representations. It is effective to distinguish entity representations from the background surroundings for recognizing camouflaged named entities. Compared with the state-of-the-art models, the results show that our method achieves competitive performance on five public datasets.

Providing Citations to Support Fact-Checking: Contextualizing Detection of Sentences Needing Citation on Small Wikipedias

Authoritative citations are critical to ensure information integrity, especially in encyclopedias like Wikipedia. To date, research on automating citation worthiness detection has largely focused on the most resourceful language, English Wikipedia, neglecting the applicability to smaller Wikipedias. In addition, previous research proposed models that analyze the content inherent to a sentence to determine its citation worthiness, overlooking the potential of additional context to improve the prediction. Addressing these gaps, our study proposes a transformer-based contextualized approach for smaller Wikipedias, presenting a novel method to compile high-quality datasets for the Albanian, Basque, and Catalan editions. We develop the Contextualized Citation Worthiness (CCW) model, employing sentence representations enriched with adjacent sentences and topic categories for enhanced contextual insight. Empirical experiments on three newly created datasets demonstrate significant performance improvements of our contextualized CCW model, with 6%, 3% and 6% absolute improvements over the baseline for Albanian, Basque and Catalan datasets, respectively. We conduct an in-depth analysis to understand the influence and extent to which preceding and succeeding context as well as topic categories contribute to the accuracy of citation-worthiness predictions. Our findings suggest that incorporating such contextual information aids in the automatic identification of sentences in need of citations, not least when both the preceding and succeeding context are incorporated. This has implications for supporting Wikipedia projects across low-resource languages, promoting better article validation and fact-checking.

Leveraging shortest dependency paths in low-resource biomedical relation extraction

BackgroundBiomedical Relation Extraction (RE) is essential for uncovering complex relationships between biomedical entities within text. However, training RE classifiers is challenging in low-resource biomedical applications with few labeled examples.MethodsWe explore the potential of Shortest Dependency Paths (SDPs) to aid biomedical RE, especially in situations with limited labeled examples. In this study, we suggest various approaches to employ SDPs when creating word and sentence representations under supervised, semi-supervised, and in-context-learning settings.ResultsThrough experiments on three benchmark biomedical text datasets, we find that incorporating SDP-based representations enhances the performance of RE classifiers. The improvement is especially notable when working with small amounts of labeled data.ConclusionSDPs offer valuable insights into the complex sentence structure found in many biomedical text passages. Our study introduces several straightforward techniques that, as demonstrated experimentally, effectively enhance the accuracy of RE classifiers.

Representations of First Language Acquisition in the Speech of Three Years Old

The study aims to represent how three-year-old children output their language acquisition. As we know, language acquisition is one of the factors in a child's development. Language acquisition which is a process in humans in obtaining the ability to understand a language that becomes the foundation in the form of sentences to communicate. This research is a descriptive qualitative research with the stages of data collection, data analysis, and data presentation. The sources and data in this study are transcriptions of recordings of verbal speech of a 3-year-old child named Hanah Shaqueena. The data has been found in this study, there are 4 types of sentences in terms of syntactic form produced by a 3-year-oldchild named Hannah Shaqueena, namely Declarative Sentence Representation, Interrogative Sentence Representation, Imperative Sentence Representation, and Exclamatory Sentence Representation.

Simple Data Transformations for Mitigating the Syntactic Similarity to Improve Sentence Embeddings at Supervised Contrastive Learning

Contrastive learning of sentence representations has achieved great improvements in several natural language processing tasks. However, the supervised contrastive learning model trained on the natural language inference (NLI) dataset is insufficient to elucidate the semantics of sentences since it is prone to make a prediction based on heuristics. Herein, by using the ParsEVAL and the word overlap metric, it is shown that sentence pairs in the NLI dataset have strong syntactic similarity and propose a framework to compensate for this problem in two aspects. 1) Apply simple syntactic transformations to the hypothesis and 2) expand the objective to SupCon Loss to leverage variants of sentences. The method is evaluated on semantic textual similarity (STS) tasks and transfer tasks. The proposed methods improve the performance of the BERT‐based baseline in STS Benchmark and SICK Relatedness by 1.48% and 2.2%. Furthermore, the model achieves 82.65% on the HANS benchmark dataset, to the best of our knowledge, which is a state‐of‐the‐art performance demonstrating that our approach is effective in grasping semantics without heuristics in the NLI dataset at supervised contrastive learning. The code is available at https://github.com/whnhch/Break-the-Similarity.

Modified R-BERT with global semantic information for relation classification task

The objective of the relation classification task is to extract relations between entities. Recent studies have found that R-BERT (Wu and He, 2019) based on pre-trained BERT (Devlin et al., 2019) acquires extremely good results in the relation classification task. However, this method does not take into account the semantic differences between different kinds of entities and global semantic information either. In this paper, we set two different fully connected layers to take into account the semantic difference between subject and object entities. Besides, we build a new module named Concat Module to fully fuse the semantic information among the subject entity vector, object entity vector, and the whole sample sentence representation vector. In addition, we apply the average pooling to acquire a better entity representation of each entity and add the activation operation with a new fully connected layer after our Concat Module. Modifying R-BERT, we propose a new model named BERT with Global Semantic Information (GSR-BERT) for relation classification tasks. We use our approach on two datasets: the SemEval-2010 Task 8 dataset and the Chinese character relationship classification dataset. Our approach achieves a significant improvement over the two datasets. It means that our approach enjoys transferability across different datasets. Furthermore, we prove that these policies we used in our approach also enjoy applicability to named entity recognition task.

Contrastive learning based on linguistic knowledge and adaptive augmentation for text classification

Pre-trained language models based on contrastive learning have shown to be effective in text classification. Despite its great success, contrastive learning still has some limitations. First, external linguistic knowledge has been shown to improve the performance of pre-trained language models, but how to use it in contrastive learning is still unclear. Second, general contrastive learning generates training samples with fixed data augmentation during the whole training period, while different augmentation methods are suitable for different downstream tasks. Fixed data augmentation can lead to suboptimal settings. In this paper, we propose contrastive learning based on linguistic knowledge and adaptive augmentation, which can obtain high-quality sentence representations to improve the performance of text classification. Specifically, we construct word-level positive and negative sample pairs by WordNet and propose a novel word-level contrastive learning function to inject linguistic knowledge. Then we dynamically select the augmentation policy by alignment and uniformity. This adaptive augmentation policy can acquire more generalized sentence representations with little computational overhead. Experiments on multiple public datasets demonstrate that our method outperforms state-of-the-art methods.