Sentence Representation Research Articles

Multi-Level Query Interaction for Temporal Language Grounding

Understanding what is happening in the surveillance video is important for human-machine interface in transportation systems, where temporal language grounding is one of the key tasks, targeting at localizing the desired moment in an untrimmed video with a given sentence query that is relevant to the moment. This task is challenging due to the following reasons: 1) the requirement of understanding the video contents and query semantics comprehensively, and 2) building the bridge between the cross-modal semantics. To tackle these problems, early methods first sample video clips and then match them with the sentence to find the most relevant one. To reduce the computational complexity associated with video clip sampling, recent methods directly predict the temporal boundaries of the desired moment on the fused features of the sentence and the video frames. However, all the previous methods often learn the word-level or phrase-level features of the sentence, or directly generates the global sentence representation by attention mechanisms or graph network. However, we argue that applying only word-level or phrase-level semantic information and cross-modal interactions is not enough to fully capture the correspondence between the video and the query. To this end, we proposed a novel Multi-level Query Exploration and Interaction (MQEI) model, which explores the semantics in both the word- and phrase-level and captures the multi-level interactions between the video and the query through an attention module. Extensive experiments on two public benchmark datasets ActivityNet Captions and Charades-STA demonstrate that the proposed model can outperform all the state-of-the-art methods consistently.

True and false recognition in MINERVA 2: Extension to sentences and metaphors

Arndt and Hirshman (1998) used MINERVA 2 to simulate true and false recognition in DRM-style lists and found that the model was able to capture many features of the empirical data. Here, we first replicate their simulations, but using empirically structured vectors derived from Latent Semantic Analysis rather than the randomly generated vectors characteristic of MINERVA 2. We report that the model still captures the DRM effect with fewer free parameters. We then extend our analyses to true and false recognition for full sentences and metaphorical expressions. Using a simple bag-of-words representation for sentences, we find that the MINERVA 2 model captures classic sentence false recognition findings from Bransford and Frank (1971) and a more recent finding from Reid and Katz (2018a) that demonstrates false recognition of unstudied sentences that share a metaphorical but not literal theme to studied sentences. These simulations provide evidence that an instance-based memory model, when amalgamated with structured semantic representations from a distributional semantic model, can account for true and false recognition across different types of language experiences.

PROFIL KEMAMPUAN PENALARAN MATEMATIKA DAN POTENSI KOGNITIF SISWA MA DALAM MENYELESAIKAN SOAL TES POTENSI SKOLASTIK (TPS)

Mathematical reasoning and cognitive potential are two student competencies assessed in the SNBT 2023 scholastic potential test. Mathematical reasoning is a student's cognitive ability in making decisions based on analysis, justification, problem solving, and evaluation of problem solving strategies based on data/information on problems solved in everyday life. The aim of the study was to obtain a profile picture of the mathematical reasoning abilities and cognitive potential of SMA students in solving TPS SNBT questions in one of the Privat SMA in East Lombok district, NTB. This research is a qualitative research with a case study design. The participants in this study were 24 students of class XII IPA SMAS Lombok Timur consisting of 8 male students and 16 female students in the odd semester of the 2022-2023 academic year. The instruments in this study were tests of mathematical reasoning and cognitive potential, interviews, observations, and documentation studies. Participant taking technique is done by purposive sampling technique. Data collection was carried out using test and non-test data. The results showed that the profile of students' mathematical reasoning ability and cognitive potential was at a high level in solving TPS SNBT questions but still found: (1) students had difficulties in understanding aspects so students could not identify and understand problems, (2) students had difficulties in modeling aspects /representation of situations so that students cannot make mathematical representations of problems, both models and other mathematical sentences, (3) students have difficulty in cognitive/concept aspects so students cannot create or come up with new strategies in solving problems, and (4) students have difficulties in aspects applied so that students are not able to make and carry out strategies and arithmetical operations correctly.

A Memory-Driven Neural Attention Model for Aspect-Based Sentiment Classification

Sentiment analysis techniques are becoming more and more important as the number of reviews on the World Wide Web keeps increasing. Aspect-based sentiment analysis (ABSA) entails the automatic analysis of sentiments at the highly fine-grained aspect level. One of the challenges of ABSA is to identify the correct sentiment expressed towards every aspect in a sentence. In this paper, a neural attention model is discussed and three extensions are proposed to this model. First, the strengths and weaknesses of the highly successful CABASC model are discussed, and three shortcomings are identified: the aspect-representation is poor, the current attention mechanism can be extended for dealing with polysemy in natural language, and the design of the aspect-specific sentence representation is upheld by a weak construction. We propose the Extended CABASC (E-CABASC) model, which aims to solve all three of these problems. The model incorporates a context-aware aspect representation, a multi-dimensional attention mechanism, and an aspect-specific sentence representation. The main contribution of this work is that it is shown that attention models can be improved upon using some relatively simple extensions, such as fusion gates and multi-dimensional attention, which can be implemented in many state-of-the-art models. Additionally, an analysis of the parameters and attention weights is provided.

Graph Convolutional Networks with POS Gate for Aspect-Based Sentiment Analysis

We make daily comments on online platforms (e.g., social networks), and such natural language texts often contain sentiment (e.g., positive and negative) for certain aspects (e.g., food and service). If we can automatically extract the aspect-based sentiment from the texts, then it will help many services or products to overcome their limitations of particular aspects. There have been studies of aspect sentiment classification (ASC) that finds sentiment towards particular aspects. Recent studies mostly adopt deep-learning models or graph neural networks as these techniques are capable of capturing linguistic patterns that contributed to performance improvement in various natural language processing tasks. In this paper, for the ASC task, we propose a new hybrid architecture of graph convolutional network (GCN) and recurrent neural network. We design a gate mechanism that jointly models word embeddings and syntactic representation of sentences. By experimental results on five datasets, we show that the proposed model outperforms other recent models and also verify that the gate mechanism contributes to the performance improvement. The overall F1 scores that we achieved is 66.64∼76.80%.

Aspect-based sentiment analysis with attention-assisted graph and variational sentence representation

Aspect-based sentiment analysis (ABSA) is a fine-grained task that detects the sentiment polarities of particular aspect words in a sentence. With the rise of graph convolution networks (GCNs), current ABSA models mostly use graph-based methods. These methods construct a dependency tree for each sentence, and regard each word as a unique node. To be more specific, they conduct classification using aspect representations instead of sentence representations, and update them with GCNs. However, this kind of method relies too much on the quality of the dependency tree and may lose the global sentence information, which is also helpful for classification. To deal with these, we design a new ABSA model AG-VSR. Two kinds of representations are proposed to perform the final classification, Attention-assisted Graph-based Representation (A2GR) and Variational Sentence Representation (VSR). A2GR is produced by the GCN module, which inputs a dependency tree modified by the attention mechanism. Furthermore, VSR is sampled from a distribution learned by a VAE-like encoder–decoder structure. Extensive experiments show that our model AG-VSR achieves competitive results. Our code and data have been released in https://github.com/wangbing1416/VAGR.

Acoustic-based automatic speech intelligibility scoring using deep neural networks

Human-generated measures of speech intelligibility are time-intensive methods for assessing the intelligibility of speech. The purpose of the present study is to automate the assessment of speech intelligibility by developing a deep neural network that estimates a standardized intelligibility score based on acoustic input. We extracted Mel-frequency cepstral coefficients from the UW/NU IEEE sentence corpus which had been manipulated with three signal-to-noise ratios (−2, 0, 2 dB). We obtained listener transcriptions from the UAW speech intelligibility dataset and calculated the Levenshtein distance between the transcriptions and the speaker's prompt. The neural network was trained to predict the Levenshtein distance given MFCC representations of sentences. We use tenfold cross-validation to verify the accuracy of the model and investigate the correlation of the model predictions with the average human responses. We also compare our model’s accuracy with the Levenshtein distance generated by transcriptions produced by the DeepSpeech ASR model. This study investigates the reliability of deep neural networks as an alternative to human-based inference in quantifying the intelligibility of speech. We discuss the advantages and disadvantages of the different approaches to assessing speech intelligibility.

Sentiment analysis based on Chinese BERT and fused deep neural networks for sentence-level Chinese e-commerce product reviews

Driven by the rapid development of Internet, more e-commerce product reviews are available on e-commerce platforms, which can help enterprises make business decisions. Currently, bidirectional encoder representations from transformers (BERT) applied in the embedding layer contributes to achieve promising results in English text sentiment analysis (SA). This paper proposes a novel model Chinese BERT with fused deep neural networks (CBERT-FDNN), extracting richer and more accurate semantic and grammatical information in Chinese text. First, Chinese BERT with whole word masking (Chinese-BERT-wwm) is used in the embedding layer to generate dynamic sentence representation vectors. It is a Chinese pre-training model based on the whole word masking (WWM) technology, which is more effective for Chinese text contextual embedding. Second, multi-channel and multi-scale convolutional neural networks (CNN) and bidirectional long short-term memory (BiLSTM) are designed to capture further crucial features in the feature extraction layer. To obtain more comprehensive sentence attributes, these features are concatenated together. Last, the model is evaluated on 100,000 sentence-level Chinese e-commerce product reviews for sentiment binary classification. The accuracy and F1 score can achieve 94.37% and 94.34%, respectively. Compared with the baseline models, the experiments show that our proposed model has higher accuracy and better prediction performance.

The role of prosodic focus in the reanalysis of garden path sentences: Depth of semantic processing impedes the revision of an erroneous local analysis

Research on post-repair representations of garden path sentences has found that readers systematically arrive at misinterpretations even after displaying evidence of reanalysis (Christianson et al., 2001; Ferreira et al., 2001). These comprehension errors have been attributed to the semantic interpretation associated with the incorrect parse persisting past disambiguation, but less is known about the mechanism driving this phenomenon (Sturt, 2007; Slattery et al., 2013). A speeded auditory comprehension experiment examined the depth of semantic processing as an independent influence on the strength of semantic persistence, drawing on known effects of pitch accent on the processing of focus-related semantic meaning (Fraundorf et al., 2010). Participants heard garden path sentences with early/late-closure ambiguity (e.g., While Anna dressed the baby stopped crying) with a sharply rising pitch accent on either the unambiguous adjunct subject or the ambiguously transitive adjunct verb, followed by a comprehension question that probed whether the incorrect late-closure analysis persisted. Since the pitch accent is often a strong cue for semantic focus when it occurs in prosodically marked phrase-medial positions, we reasoned that a deeper semantic processing would be facilitated for the late-closure analysis only when the verb receives a pitch accent. Findings indicate that a pitch accent on the verb significantly decreased accuracy without a corresponding increase on response time, suggesting that a deeper semantic processing of the erroneous parse can strengthen its resistance to revision without necessarily interfering with the process of structure-building.

EANT: Distant Supervision for Relation Extraction with Entity Attributes via Negative Training

Distant supervision for relation extraction (DSRE) automatically acquires large-scale annotated data by aligning the corpus with the knowledge base, which dramatically reduces the cost of manual annotation. However, this technique is plagued by noisy data, which seriously affects the model’s performance. In this paper, we introduce negative training to filter them out. Specifically, we train the model with the complementary label based on the idea that “the sentence does not express the target relation”. The trained model can discriminate the noisy data from the training set. In addition, we believe that additional entity attributes (such as description, alias, and types) can provide more information for sentence representation. On this basis, we propose a DSRE model with entity attributes via negative training called EANT. While filtering noisy sentences, EANT also relabels some false negative sentences and converts them into useful training data. Our experimental results on the widely used New York Times dataset show that EANT can significantly improve the relation extraction performance over the state-of-the-art baselines.

Specific Relation Attention-Guided Graph Neural Networks for Joint Entity and Relation Extraction in Chinese EMR

Electronic medical records (EMRs) contain a variety of valuable medical entities and their relations. The extraction of medical entities and their relations has important application value in the structuring of EMR and the development of various types of intelligent assistant medical systems, and hence is a hot issue in intelligent medicine research. In recent years, most research aims to firstly identify entities and then to recognize the relations between the entities, and often suffers from many redundant operations. Furthermore, the challenge remains of identifying overlapping relation triplets along with the entire medical entity boundary and detecting multi-type relations. In this work, we propose a Specific Relation Attention-guided Graph Neural Networks (SRAGNNs) model to jointly extract entities and their relations in Chinese EMR, which uses sentence information and attention-guided graph neural networks to perceive the features of every relation in a sentence and then to extract those relations. In addition, a specific sentence representation is constructed for every relation, and sequence labeling is performed to extract its corresponding head and tail entities. Experiments on a medical evaluation dataset and a manually labeled Chinese EMR dataset show that our model improves the performance of Chinese medical entities and relation extraction.

A Two-channel model for relation extraction using multiple trained word embeddings

As an essential task in the field of knowledge graph, relation extraction (RE) has received extensive attention from researchers. Since the existing RE methods only adopt one trained word embedding to obtain sentence representation, the polysemy problem cannot be well solved. In order to alleviate the polysemy in RE, this paper proposes a Two-channel model by adopting multiple trained word embeddings, in which one channel is a bidirectional long-short-term memory network based on an attention mechanism (Bi-LSTM-ATT), and the other channel is a convolutional neural network (CNN). Furthermore, a two-channel fusion method is proposed based on this model to deal with polysemy problem in RE. As a result, the Two-channel model achieves 85.42% and 62.2% F1-scores on the Semeval-2010 Task 8 dataset and KBP37 dataset, respectively. The experiment results show that the Two-channel model performs better than most existing models under the condition without using the external features generated by natural language processing (NLP) tools. On the other hand, the two-channel fusion method also obtains a better performance than either concatenation or addition on the two channels.

BRFP: An Efficient and Universal Sentence Embedding Learning Model Method Supporting Fused Syntax Combined with Graph Embedding Representation Algorithm

Due to the rapidly growing volume of data on the Internet, the methods of efficiently and accurately processing massive text information have been the focus of research. In natural language processing theory, sentence embedding representation is an important method. This paper proposes a new sentence embedding learning model called BRFP (Factorization Process with Bidirectional Restraints) that fuses syntactic information, uses matrix decomposition to learn syntactic information, and fuses and calculates with word vectors to obtain the embedded representation of sentences. In the experimental chapter, text similarity experiments are conducted to verify the rationality and effectiveness of the model and analyzed experimental results on Chinese and English texts with the current mainstream learning methods, and potential improvement directions are summarized. The experimental results on Chinese and English datasets, including STS, AFQMC, and LCQMC, show that the model proposed in this paper outperforms the CNN method in terms of accuracy and F1 value by 7.6% and 4.8. The comparison experiment with the word vector weighted model shows that when the sentence length is longer, or the corresponding syntactic structure is complex, the model’s advantages in this paper are more prominent than TF-IDF and SIF methods. Compared with the TF-IDF method, the effect improved by 14.4%. Compared with the SIF method, it has a maximum advantage of 7.9%, and the overall improvement in each comparative experimental task is between 4 and 6 percentage points. In the neural network model comparison experiment, the model in this paper compared the CNN, RNN, LSTM, ST, QT, and InferSent models, and the effect significantly improved on the 14’OnWN, 14’Tweet-news, and 15’Ans.-forum datasets. For example, in the 14’OnWN dataset, the BRFP method has a 10.9% improvement over the ST method. The 14’Tweet-news dataset has a 22.9% advantage over the LSTM method, and the 15’Ans.-forum dataset has a 24.07% improvement over the RNN method. The article also demonstrates the generality of the model, proving that the model proposed in this paper is also a universal learning framework.

Improving extractive document summarization with sentence centrality

Extractive document summarization (EDS) is usually seen as a sequence labeling task, which extracts sentences from a document one by one to form a summary. However, extracting sentences separately ignores the relationship between the sentences and documents. One solution is to use sentence position information to enhance sentence representation, but this will cause the sentence-leading bias problem, especially in news datasets. In this paper, we propose a novel sentence centrality for the EDS task to address these two problems. The sentence centrality is based on directed graphs, while reflecting the sentence-document relationship, it also reflects the sentence position information in the document. We implicitly strengthen the relevance of sentences and documents by using sentence centrality to enhance sentence representation. Notably, we replaced the sentence position information with sentence centrality to reduce sentence-leading bias without causing model performance degradation. Experiments on the CNN/Daily Mail dataset showed that EDS models with sentence centrality significantly improved compared with baseline models.

A commonality-based enhancement for sentence modeling with supervision

Sentence pair modeling is a fundamental yet challenging issue for feature mining in natural language processing (NLP) tasks. Recently, most works have generated feature and sentence representation based on the interactive attention mechanism. However, these models have two limitations: (1) they only consider global information through attention coefficient weighting, which makes insufficient utilization of critical features; (2) they only conduct internal training by fine-tuning network parameters, in which attention results are poorly explained. In this paper, inspired by human reasoning, we propose a Commonality Aggregated approach (CA) to enhance the lightweight interaction model by considering phrase features and contextual words. Specifically, we first fuse positional encoding and employ supervised training to extract critical phrase information from the text as the commonality of sentence pairs. Then, we deploy transfer learning and utilize interaction network to combine crucial phrase features, core word features, and positional encoding to enhance sentence pair modeling. Compared with the original network, extensive experiments on multiple benchmark datasets demonstrate the effectiveness of the proposed commonality aggregated method with stronger competitiveness. Further visual analysisanalysies validated the more explicit interpretability of attention, and extended experimental results indicate the excellent generalization of our approach.

Sentence part-enhanced BERT with respect to downstream tasks

Bidirectional encoder representations from transformers (BERT) have achieved great success in many natural language processing tasks. However, BERT generally takes the embedding of the first token to represent sentence meaning in the tasks such as sentiment analysis and textual similarity, which does not properly treat different sentence parts. Different sentence parts have different levels of importance for different downstream tasks. For example, main parts (subject, predicate, and object) play crucial roles in textual similarity calculation, while secondary parts (adverbial and complement) are more important than the main parts in sentiment analysis. To this end, we propose a sentence part-enhanced BERT (SpeBERT) model that uses sentence parts with respect to downstream tasks to enhance sentence representations. Specifically, we encode sentence parts based on dependency parsing and downstream tasks, and extract embeddings through a pooling operation. Furthermore, we design several fusion strategies to incorporate different embeddings. We evaluate the proposed SpeBERT model on two downstream tasks, sentiment classification, and semantic textual similarity, with six benchmark datasets. The experimental results show that our model achieves better performance than competitor models.

Improving Abusive Language Detection with online interaction network

The rapid development of online social media makes Abusive Language Detection (ALD) a hot topic in the field of affective computing. However, most methods for ALD in social networks do not take into account the interactive relationships among user posts, which simply regard ALD as a task of text context representation learning. To solve this problem, we propose a pipeline approach that considers both the context of a post and the characteristics of interaction network in which it is posted. Specifically, our method is divided into pre-training and downstream tasks. First, to capture fine contextual features of the posts, we use Bidirectional Encoder Representation from Transformers (BERT) as Encoder to generate sentence representations. Later, we build a Relation-Special Network according to the semantic similarity between posts as well as the interaction network structural information. On this basis, we design a Relation-Special Graph Neural Network (RSGNN) to spread effective information in the interaction network and learn the classification of texts. The experiment proves that our method can effectively improve the detection effect of abusive posts over three public datasets. The results demonstrate that injecting interaction network structure into the abusive language detection task can significantly improve the detection results.

CharAs-CBert: Character Assist Construction-Bert Sentence Representation Improving Sentiment Classification.

In the process of semantic capture, traditional sentence representation methods tend to lose a lot of global and contextual semantics and ignore the internal structure information of words in sentences. To address these limitations, we propose a sentence representation method for character-assisted construction-Bert (CharAs-CBert) to improve the accuracy of sentiment text classification. First, based on the construction, a more effective construction vector is generated to distinguish the basic morphology of the sentence and reduce the ambiguity of the same word in different sentences. At the same time, it aims to strengthen the representation of salient words and effectively capture contextual semantics. Second, character feature vectors are introduced to explore the internal structure information of sentences and improve the representation ability of local and global semantics. Then, to make the sentence representation have better stability and robustness, character information, word information, and construction vectors are combined and used together for sentence representation. Finally, the evaluation and verification are carried out on various open-source baseline data such as ACL-14 and SemEval 2014 to demonstrate the validity and reliability of sentence representation, namely, the F1 and ACC are 87.54% and 92.88% on ACL14, respectively.

Bi-GRU Relation Extraction Model Based on Keywords Attention

Abstract Relational extraction plays an important role in the field of natural language processing to predict semantic relationships between entities in a sentence. Currently, most models have typically utilized the natural language processing tools to capture high-level features with an attention mechanism to mitigate the adverse effects of noise in sentences for the prediction results. However, in the task of relational classification, these attention mechanisms do not take full advantage of the semantic information of some keywords which have information on relational expressions in the sentences. Therefore, we propose a novel relation extraction model based on the attention mechanism with keywords, named Relation Extraction Based on Keywords Attention (REKA). In particular, the proposed model makes use of bi-directional GRU (Bi-GRU) to reduce computation, obtain the representation of sentences, and extracts prior knowledge of entity pair without any NLP tools. Besides the calculation of the entity-pair similarity, Keywords attention in the REKA model also utilizes a linear-chain conditional random field (CRF) combining entity-pair features, similarity features between entity-pair features, and its hidden vectors, to obtain the attention weight resulting from the marginal distribution of each word. Experiments demonstrate that the proposed approach can utilize keywords incorporating relational expression semantics in sentences without the assistance of any high-level features and achieve better performance than traditional methods.

Unsupervised Sentence Representation via Contrastive Learning with Mixing Negatives

Unsupervised sentence representation learning is a fundamental problem in natural language processing. Recently, contrastive learning has made great success on this task. Existing constrastive learning based models usually apply random sampling to select negative examples for training. Previous work in computer vision has shown that hard negative examples help contrastive learning to achieve faster convergency and better optimization for representation learning. However, the importance of hard negatives in contrastive learning for sentence representation is yet to be explored. In this study, we prove that hard negatives are essential for maintaining strong gradient signals in the training process while random sampling negative examples is ineffective for sentence representation. Accordingly, we present a contrastive model, MixCSE, that extends the current state-of-the-art SimCSE by continually constructing hard negatives via mixing both positive and negative features. The superior performance of the proposed approach is demonstrated via empirical studies on Semantic Textual Similarity datasets and Transfer task datasets.