Sentence-level Representation Research Articles

Few-shot intent detection with self-supervised pretraining and prototype-aware attention

Few-shot intent detection is a more challenging application. However, traditional prototypical networks based on averaging often suffer from issues such as missing key information, poor generalization capabilities. In previous work, using three-dimensional convolutional neural networks (3DCNN) to generate prototype representations faces challenges with long-distance dependencies. Furthermore, a pretrained encoder’s performance in a specific domain is often suboptimal because its knowledge of the specific domain is fragmented. Therefore, in this paper, we propose a simple yet effective two-stage learning strategy to address these issues. In the first stage, we propose a self-supervised multi-task pretraining (SMTP) strategy. SMTP utilizes unlabeled data from the current domain to help the pretrained encoder learn the semantic information of the text and implicitly distinguish semantically similar text representations without using any labels. SMTP aims to enhance the representation capability of the pretrained encoder in a specific domain. In the second stage, we propose a prototype-aware attention (PaAT) model to generate prototype representations of the same class. PaAT generates prototype representations by calculating the attention between class texts, which can effectively solve the long-distance dependence problem of 3DCNN. PaAT is a siamese architecture that can simultaneously generate prototype representations and sentence-level representations of unseen data. In addition, to prevent overfitting in few-shot learning, we introduce an unsupervised contrastive regularization term to constrain PaAT. Our method achieves state-of-the-art performance on four public datasets.11Our code is available: https://github.com/YS19999/SMTP-PaAT. .

Sentence superiority in the reading brain

When a sequence of written words is briefly presented and participants are asked to identify just one word at a post-cued location, then word identification accuracy is higher when the word is presented in a grammatically correct sequence compared with an ungrammatical sequence. This sentence superiority effect has been reported in several behavioral studies and two EEG investigations. Taken together, the results of these studies support the hypothesis that the sentence superiority effect is primarily driven by rapid access to a sentence-level representation via partial word identification processes that operate in parallel over several words. Here we used MEG to examine the neural structures involved in this early stage of written sentence processing, and to further specify the timing of the different processes involved. Source activities over time showed grammatical vs. ungrammatical differences first in the left inferior frontal gyrus (IFG: 321–406 ms), then the left anterior temporal lobe (ATL: 466–531 ms), and finally in both left IFG (549–602 ms) and left posterior superior temporal gyrus (pSTG: 553–622 ms). We interpret the early IFG activity as reflecting the rapid bottom-up activation of sentence-level representations, including syntax, enabled by partly parallel word processing. Subsequent activity in ATL and pSTG is thought to reflect the constraints imposed by such sentence-level representations on on-going word-based semantic activation (ATL), and the subsequent development of a more detailed sentence-level representation (pSTG). These results provide further support for a cascaded interactive-activation account of sentence reading.

Word- and Sentence-Level Representations for Implicit Aspect Extraction

Word- and Sentence-Level Representations for Implicit Aspect Extraction

BioEGRE: a linguistic topology enhanced method for biomedical relation extraction based on BioELECTRA and graph pointer neural network

BackgroundAutomatic and accurate extraction of diverse biomedical relations from literature is a crucial component of bio-medical text mining. Currently, stacking various classification networks on pre-trained language models to perform fine-tuning is a common framework to end-to-end solve the biomedical relation extraction (BioRE) problem. However, the sequence-based pre-trained language models underutilize the graphical topology of language to some extent. In addition, sequence-oriented deep neural networks have limitations in processing graphical features.ResultsIn this paper, we propose a novel method for sentence-level BioRE task, BioEGRE (BioELECTRA and Graph pointer neural net-work for Relation Extraction), aimed at leveraging the linguistic topological features. First, the biomedical literature is preprocessed to retain sentences involving pre-defined entity pairs. Secondly, SciSpaCy is employed to conduct dependency parsing; sentences are modeled as graphs based on the parsing results; BioELECTRA is utilized to generate token-level representations, which are modeled as attributes of nodes in the sentence graphs; a graph pointer neural network layer is employed to select the most relevant multi-hop neighbors to optimize representations; a fully-connected neural network layer is employed to generate the sentence-level representation. Finally, the Softmax function is employed to calculate the probabilities. Our proposed method is evaluated on three BioRE tasks: a multi-class (CHEMPROT) and two binary tasks (GAD and EU-ADR). The results show that our method achieves F1-scores of 79.97% (CHEMPROT), 83.31% (GAD), and 83.51% (EU-ADR), surpassing the performance of existing state-of-the-art models.ConclusionThe experimental results on 3 biomedical benchmark datasets demonstrate the effectiveness and generalization of BioEGRE, which indicates that linguistic topology and a graph pointer neural network layer explicitly improve performance for BioRE tasks.

A Contextual Dependency-Aware Graph Convolutional Network for extracting entity relations

Dependency trees reflect rich structural information, which can effectively guide models to understand text semantics and are widely used for relation extraction. However, existing dependency-based models suffer from noise in dependency trees, which distorts the modeling of contextual information, particularly when introducing dependency types, further exacerbating the propagation and accumulation of errors. In addition, rule-based pruning strategies used to eliminate noise in dependency trees may lose crucial information for capturing relational patterns, whereas unconstrained learning-based pruning strategies may introduce more noisy edges. These two aspects of dependency trees may cause models to produce semantic confusion, which is detrimental to relation extraction. In this study, we propose a Contextual Dependency-Aware Graph Convolutional Network (C-DAGCN) that works on complete dependency trees and perceives the importance of word dependencies from multiple angles without considering the noise in dependency trees. This study includes three parts. First, the sequence model captures semantic information for the input sentence and its words. We considered the appropriate outputs as word and sentence-level representations based on model characteristics. Second, the structure model enhances the description of the sentence structure and its guidance. We consider that position knowledge implies associations between words; therefore, we propose the Current Word Global Relative Position Knowledge (CW-GRPK) to globally enhance text structure modeling and reduce the models’ overreliance on single noisy structure guidance. A dependency-aware module weighs heuristic syntactic knowledge and non-heuristic CW-GRPK to capture fine-grained word-level interactions that effectively distinguish words related to relational semantics. Subsequently, a dependency-guided module, based on the results of the dependency-aware layer, encodes a knowledge-enhanced word graph without losing global linguistic patterns. Third, we constructed high-level linguistic representations based on the results of the sequence and structure model in C-DAGCN as a relational representation and mapped it to the relational decision space for relational prediction. The experimental results demonstrate that our method outperforms nine strong baseline models on two benchmark datasets. Compared with the best baseline Attentive Graph Convolutional Networks (A-GCN) model (Tian et al., 2021), our proposed C-DAGCN model with 90.15 macro-average F1 and 79.52 micro-average F1 achieves 0.28 macro-average F1 and 0.47 micro-average F1 improvement on the SemEval2010-Task8 and ACE2005 datasets, respectively. 11Our code is available at https://github.com/dymyyc/RE_C-DAGCN. The released code when run produces results which are the same as those reported in this paper.

Improving document-level event detection with event relation graph

The correlation between events within the same document plays a crucial role in event detection. Most existing detection models often ignore event correlations, which is not applicable to multi-event detection at the document level. In the real world, it is a common phenomenon that the probability of correlated events occurring simultaneously is much greater than the probability of uncorrelated events occurring simultaneously. Based on this observation, we propose an event correlation-based document-level event detection model (EventCo-ED) to capture the document-level association between events. Specifically, EventCo-ED first constructs a novel event relation graph (ERG) to capture the correlation between events and uses this correlation to extract the topic features of a document. Secondly, DMBERT is employed to get sentence-level contextual representation as the local features. Finally, a gated feature fusion module is used to aggregate topic features and local features, and a correlation suppression module is used to increase the probability that related events are detected simultaneously and suppress the probability that unrelated events are detected simultaneously. Experimental results show that the proposed model can simultaneously improve the precision and recall of multi-event detection and achieve 1.56% and 3.63% F1 improvements on the LEVEN and MAVEN corpuses, respectively.

Is beta in agreement with the relatives? Using relative clause sentences to investigate MEG beta power dynamics during sentence comprehension.

There remains some debate about whether beta power effects observed during sentence comprehension reflect ongoing syntactic unification operations (beta-syntax hypothesis), or instead reflect maintenance or updating of the sentence-level representation (beta-maintenance hypothesis). In this study, we used magnetoencephalography to investigate beta power neural dynamics while participants read relative clause sentences that were initially ambiguous between a subject- or an object-relative reading. An additional condition included a grammatical violation at the disambiguation point in the relative clause sentences. The beta-maintenance hypothesis predicts a decrease in beta power at the disambiguation point for unexpected (and less preferred) object-relative clause sentences and grammatical violations, as both signal a need to update the sentence-level representation. While the beta-syntax hypothesis also predicts a beta power decrease for grammatical violations due to a disruption of syntactic unification operations, it instead predicts an increase in beta power for the object-relative clause condition because syntactic unification at the point of disambiguation becomes more demanding. We observed decreased beta power for both the agreement violation and object-relative clause conditions in typical left hemisphere language regions, which provides compelling support for the beta-maintenance hypothesis. Mid-frontal theta power effects were also present for grammatical violations and object-relative clause sentences, suggesting that violations and unexpected sentence interpretations are registered as conflicts by the brain's domain-general error detection system.

Cortical Tracking of Continuous Speech Under Bimodal Divided Attention.

Speech processing often occurs amid competing inputs from other modalities, for example, listening to the radio while driving. We examined the extent to which dividing attention between auditory and visual modalities (bimodal divided attention) impacts neural processing of natural continuous speech from acoustic to linguistic levels of representation. We recorded electroencephalographic (EEG) responses when human participants performed a challenging primary visual task, imposing low or high cognitive load while listening to audiobook stories as a secondary task. The two dual-task conditions were contrasted with an auditory single-task condition in which participants attended to stories while ignoring visual stimuli. Behaviorally, the high load dual-task condition was associated with lower speech comprehension accuracy relative to the other two conditions. We fitted multivariate temporal response function encoding models to predict EEG responses from acoustic and linguistic speech features at different representation levels, including auditory spectrograms and information-theoretic models of sublexical-, word-form-, and sentence-level representations. Neural tracking of most acoustic and linguistic features remained unchanged with increasing dual-task load, despite unambiguous behavioral and neural evidence of the high load dual-task condition being more demanding. Compared to the auditory single-task condition, dual-task conditions selectively reduced neural tracking of only some acoustic and linguistic features, mainly at latencies >200 ms, while earlier latencies were surprisingly unaffected. These findings indicate that behavioral effects of bimodal divided attention on continuous speech processing occur not because of impaired early sensory representations but likely at later cognitive processing stages. Crossmodal attention-related mechanisms may not be uniform across different speech processing levels.

Chunk-Level Speech Emotion Recognition: A General Framework of Sequence-to-One Dynamic Temporal Modeling

A critical issue of current speech-based sequence-to-one learning tasks, such as speech emotion recognition(SER), is the dynamic temporal modeling for speech sentences with different durations. The goal is to extract an informative representation vector of the sentence from acoustic feature sequences with varied length. Traditional methods rely on static descriptions such as statistical functions or a universal background model (UBM), which are not capable of characterizing dynamic temporal changes. Recent advances in deep learning architectures provide promising results, directly extracting sentence-level representations from frame-level features. However, conventional cropping and padding techniques that deal with varied length sequences are not optimal, since they truncate or artificially add sentence-level information. Therefore, we propose a novel dynamic chunking approach, which maps the original sequences of different lengths into a fixed number of chunks that have the same duration by adjusting their overlap. This simple chunking procedure creates a flexible framework that can incorporate different feature extractions and sentence-level temporal aggregation approaches to cope, in a principled way, with different sequence-to-one tasks. Our experimental results based on three databases demonstrate that the proposed framework provides: 1) improvement in recognition accuracy, 2) robustness toward different temporal length predictions, and 3) high model computational efficiency advantages.

DEEDP: Document-Level Event Extraction Model Incorporating Dependency Paths

Document-level event extraction (DEE) aims at extracting event records from given documents. Existing DEE methods handle troublesome challenges by using multiple encoders and casting the task into a multi-step paradigm. However, most of the previous approaches ignore a missing feature by using mean pooling or max pooling operations in different encoding stages and have not explicitly modeled the interdependency features between input tokens, and thus the long-distance problem cannot be solved effectively. In this study, we propose Document-level Event Extraction Model Incorporating Dependency Paths (DEEDP), which introduces a novel multi-granularity encoder framework to tackle the aforementioned problems. Specifically, we first designed a Transformer-based encoder, Transformer-M, by adding a Syntactic Feature Attention mechanism to the Transformer, which can capture more interdependency information between input tokens and help enhance the semantics for sentence-level representations of entities. We then stacked Transformer-M and Transformer to integrate sentence-level and document-level features; we thus obtained semantic enhanced document-aware representations for each entity and model long-distance dependencies between arguments. Experimental results on the benchmarks MUC-4 and ChFinAnn demonstrate that DEEDP achieves superior performance over the baselines, proving the effectiveness of our proposed methods.

Attentional Extractive Summarization

In this work, a general theoretical framework for extractive summarization is proposed—the Attentional Extractive Summarization framework. Although abstractive approaches are generally used in text summarization today, extractive methods can be especially suitable for some applications, and they can help with other tasks such as Text Classification, Question Answering, and Information Extraction. The proposed approach is based on the interpretation of the attention mechanisms of hierarchical neural networks, which compute document-level representations of documents and summaries from sentence-level representations, which, in turn, are computed from word-level representations. The models proposed under this framework are able to automatically learn relationships among document and summary sentences, without requiring Oracle systems to compute the reference labels for each sentence before the training phase. These relationships are obtained as a result of a binary classification process, the goal of which is to distinguish correct summaries for documents. Two different systems, formalized under the proposed framework, were evaluated on the CNN/DailyMail and the NewsRoom corpora, which are some of the reference corpora in the most relevant works on text summarization. The results obtained during the evaluation support the adequacy of our proposal and suggest that there is still room for the improvement of our attentional framework.

Learning Sentence-Level Representations with Predictive Coding

Learning sentence representations is an essential and challenging topic in the deep learning and natural language processing communities. Recent methods pre-train big models on a massive text corpus, focusing mainly on learning the representation of contextualized words. As a result, these models cannot generate informative sentence embeddings since they do not explicitly exploit the structure and discourse relationships existing in contiguous sentences. Drawing inspiration from human language processing, this work explores how to improve sentence-level representations of pre-trained models by borrowing ideas from predictive coding theory. Specifically, we extend BERT-style models with bottom-up and top-down computation to predict future sentences in latent space at each intermediate layer in the networks. We conduct extensive experimentation with various benchmarks for the English and Spanish languages, designed to assess sentence- and discourse-level representations and pragmatics-focused assessments. Our results show that our approach improves sentence representations consistently for both languages. Furthermore, the experiments also indicate that our models capture discourse and pragmatics knowledge. In addition, to validate the proposed method, we carried out an ablation study and a qualitative study with which we verified that the predictive mechanism helps to improve the quality of the representations.

Hierarchical Sliding Inference Generator for Question-driven Abstractive Answer Summarization

Text summarization on non-factoid question answering (NQA) aims at identifying the core information of redundant answer guidance using questions, which can dramatically improve answer readability and comprehensibility. Most existing approaches focus on extracting query-related sentences to construct a summary, where the logical connection of natural language and the hierarchical interpretable semantic association are often neglected, thus degrading performance. To address these issues, we propose a novel question-driven abstractive answer summarization model, called the H ierarchical S liding I nference G enerator (HSIG), to form inferable and interpretable summaries by explicitly introducing hierarchical information reasoning between questions and corresponding answers. Specifically, we first apply an elaborately designed hierarchical sliding fusion inference model to determine the most relevant question sentence-level representation that provides a deeper interpretable basis for sentence selection in summarization, which further increases computational performance on the premise of following the semantic inheritance structure. Additionally, to improve summary fluency, we construct a double-driven selective generator to integrate various semantic information from two mutual question-and-answer perspectives. Experimental results illustrate that compared with state-of-the-art baselines, our model achieves remarkable improvement on two benchmark datasets and specifically improves the 2.46 ROUGE-1 points on PubMedQA, which demonstrates the superiority of our model on abstractive summarization with hierarchical sequential reasoning.

MSRL-Net: A multi-level semantic relation-enhanced learning network for aspect-based sentiment analysis

Aspect-based sentiment analysis (ABSA) aims to analyze the sentiment polarity of a given text towards several specific aspects. For implementing the ABSA, one way is to convert the original problem into a sentence semantic matching task, using pre-trained language models, such as BERT. However, for such a task, the intra- and inter-semantic relations among input sentence pairs are often not considered. Specifically, the semantic information and guidance of relations revealed in the labels, such as positive, negative and neutral, have not been completely exploited. To address this issue, we introduce a self-supervised sentence pair relation classification task and propose a model named Multi-level Semantic Relation-enhanced Learning Network (MSRL-Net) for ABSA. In MSRL-Net, after recasting the original ABSA task as a sentence semantic matching task, word dependency relations and word-sentence relations are utilized to enhance the word-level semantic representation for the sentence semantic matching task, while sentence semantic relations and sentence pairs relations are utilized to enhance the sentence-level semantic representation for sentence pair relation classification. Empirical experiments on SemEval 2014 Task 4, SemEval 2016 Task 5 and SentiHood show that MSRL-Net significantly outperforms other baselines such as BERT in terms of accuracy, Macro-F1 and AUC.

On Robustness and Sensitivity of a Neural Language Model: A Case Study on Italian L1 Learner Errors

In this paper, we propose a comprehensive linguistic study aimed at assessing the implicit behavior of one of the most prominent Neural Language Models (NLM) based on Transformer architectures, BERT Devlin et al., when dealing with a particular source of noisy data, namely essays written by L1 Italian learners containing a variety of errors targeting grammar, orthography and lexicon. Differently from previous works, we focus on the pre-training stage and we devise two complementary evaluation tasks aimed at assessing the impact of errors on sentence-level inner representations in terms of semantic robustness and linguistic sensitivity. While the first evaluation perspective is meant to probe the model's ability to encode the semantic similarity between sentences also in the presence of errors, the second type of probing task evaluates the influence of errors on BERT's implicit knowledge of a set of raw and morpho-syntactic properties of a sentence. Our experiments show that BERT's ability to compute sentence similarity and to correctly encode multi-leveled linguistic information of a sentence are differently modulated by the category of errors and that the error hierarchies in terms of robustness and sensitivity change across layer-wise representations.

Word-level dual channel with multi-head semantic attention interaction for community question answering

<abstract><p>The semantic matching problem detects whether the candidate text is related to a specific input text. Basic text matching adopts the method of statistical vocabulary information without considering semantic relevance. Methods based on Convolutional neural networks (CNN) and Recurrent networks (RNN) provide a more optimized structure that can merge the information in the entire sentence into a single sentence-level representation. However, these representations are often not suitable for sentence interactive learning. We design a multi-dimensional semantic interactive learning model based on the mechanism of multiple written heads in the transformer architecture, which not only considers the correlation and position information between different word levels but also further maps the representation of the sentence to the interactive three-dimensional space, so as to solve the problem and the answer can select the best word-level matching pair, respectively. Experimentally, the algorithm in this paper was tested on Yahoo! and StackEx open-domain datasets. The results show that the performance of the method proposed in this paper is superior to the previous CNN/RNN and BERT-based methods.</p></abstract>

Research On Text Classification Based On Deep Neural Network

Text classification is one of the classic tasks in the field of natural language processing. The goal is to identify the category to which the text belongs. Text categorization is widely used in email detection, sentiment analysis, topic marking and other fields. However, good text representation is the key to improve the performance of natural language processing tasks such as text classification. Traditional text representation adopts bag-of-words model or vector space model, which not only loses the context information of the text, but also faces the problems of high latitude and high sparsity. In recent years, with the increase of data and the improvement of computing performance, the use of deep learning technology to represent and classify texts has attracted great attention. Convolutional neural network, recurrent neural network and recurrent neural network with attention mechanism are used to represent the text, and then to classify the text and other natural language processing tasks, all of which have better performance than the traditional methods. In this paper, we design two sentence-level text representation and classification models based on the deep network. The details are as follows: (1) Text representation and classification model based on bidirectional cyclic and convolutional neural networks-BRCNN. Brcnn's input is the word vector corresponding to each word in the sentence; After using cyclic neural network to extract word order information in sentences, convolution neural network is used to extract higher-level features of sentences. After convolution, the maximum pool operation is used to obtain sentence vectors. At last, softmax classifier is used for classification. Cyclic neural network can capture the word order information in sentences, while convolutional neural network can extract useful features. Experiments on eight text classification tasks show that BRCNN model can get better text feature representation, and the classification accuracy rate is equal to or higher than that of the prior art.. (2) A text representation and classification model based on attention mechanism and convolutional neural network-ACNN. ACNN model uses the recurrent neural network with attention mechanism to obtain the context vector; Then convolution neural network is used to extract more advanced feature information. The maximum pool operation is adopted to obtain a sentence vector; At last, the softmax classifier is used to classify the text. Experiments on eight text classification benchmark data sets show that ACNN improves the stability of model convergence, and can converge to an optimal or local optimal solution better than BRCNN.

Exploiting global context and external knowledge for distantly supervised relation extraction

Distantly supervised relation extraction aims to obtain relational facts from unstructured texts. Although distant supervision can automatically generate labeled training instances, it inevitably suffers from the wrong-label problem. Most of the current work is based on the bag-level for solving the noise problem, where a bag is composed of multiple sentences containing mentions of the same entity pair. However, previous studies mostly represent sentences from a single perspective, wherein insufficient modeling of global information restricts the effectiveness of denoising. In this study, we propose a novel distantly supervised relation extraction approach that incorporates the global contextual information of sentences to guide the denoising process and generate an effective bag-level representation. Simultaneously, knowledge-aware word embeddings were generated to enrich sentence-level representations by introducing both structured knowledge from external knowledge graphs and semantic knowledge from the corpus. The experimental results demonstrate that our proposed approach outperforms state-of-the-art methods on both versions of the large-scale benchmark New York Times dataset. In addition, the differences between the two versions of the dataset were investigated through further comparative experiments.

A Bi-level representation learning model for medical visual question answering

Medical Visual Question Answering (VQA) targets at answering questions related to given medical images and it contains tremendous potential in healthcare services. However, researches on medical VQA are still facing challenges, particularly on how to learn a fine-grained multimodal semantic representation from relatively small volume of data resources for answer prediction. Moreover, the long-tailed distribution labels of medical VQA data frequently result in poor performance of models. To this end, we propose a novel bi-level representation learning model with two reasoning modules to learn bi-level representations for the medical VQA task. One is sentence-level reasoning to learn sentence-level semantic representations from multimodal input. The other is token-level reasoning that employs an attention mechanism to generate a multimodal contextual vector by fusing image features and word embeddings. The contextual vector is used to filter irrelevant semantic representations from sentence-level reasoning to generate a fine-grained multimodal representation. Furthermore, a label-distribution-smooth margin loss is proposed to minimize generalization error bound of long-tailed distribution datasets by modifying margin bound of different labels in training set. Based on standard VQA-Rad dataset and PathVQA dataset, the proposed model achieves 0.7605 and 0.5434 on accuracy, 0.7741 and 0.5288 on F1-score, respectively, outperforming a set of state-of-the-art baseline models.

Contrastive semantic similarity learning for image captioning evaluation

Automatically evaluating the quality of image captions can be very challenging since human language is quite flexible that there can be various expressions for the same meaning. Most current captioning metrics rely on token-level matching between candidate caption and the ground truth label sentences. It usually neglects the sentence-level information. Motivated by the auto-encoder mechanism and contrastive representation learning advances, we propose a learning-based metric I2CE (Intrinsic Image Captioning Evaluation). For learning the evaluation metric, we develop three progressive model structures capturing the sentence level representations–single branch model, dual branches model, and triple branches model. For evaluation of the proposed metric, we select one automatic captioning model and collect human scores on the quality of the generated captions. We introduce a statistical test on the correlation between human scores and metric scores. Our proposed metric I2CE achieves the Spearman correlation value of 51.42, which is better than the score of 41.95 achieved by one recently proposed BERT-based metric. The result is also better than the conventional rule-based metrics. Extensive results on the Composite-coco dataset and PASCAL-50S also validate the effectiveness of our proposed metric. The proposed metric could serve as a novel indicator of the intrinsic information between captions, which complements the existing ones.