SemEval Datasets Research Articles

HỌC BIỂU DIỄN CÂU SỬ DỤNG MÔ HÌNH LSTM TRONG BÀI TOÁN TÌM KIẾM CÂU HỎI

Learning sentence representation with the full semantics of a document is a challenge in natural language processing problems because if the semantic representation vector of the sentence is suitable, it will increase the performance of finding similar question problems. In this paper, we propose implementing a series of LSTM models with different ways of extracting sentence representations and applying them to question retrieval to exploit the hidden semantics of sentences. These methods give sentence representation from hidden layers of the LSTM model. The results show that the technique using a combination of both Max Pooling and Mean Pooling gives the highest results on the 2017 SemEval dataset for the problem of finding similarity questions. Keywords: LSTM; Deep Learning; NLP; QA; learning sentence representation; CQA.

Self-augmented sequentiality-aware encoding for aspect term extraction

Aspect Term Extraction (ATE) is a natural language processing task which recognizes and extracts aspect terms from sentences. The recent study in this field successfully leverages Pretrained Language Model (PLM) and data augmentation, which contributes to the construction of knowledgeable encoder for ATE. In this paper, we propose a novel method to strengthen ATE encoder, using self-augmented mechanism and sequentiality-aware network. Specifically, self augmentation expands training data with paraphrases and fuses correlated information using PLM during encoding. Back translation is used for obtaining pragmatically-diverse paraphrases. On the basis, BiGRU is utilized as a supplementary neural layer for extra-encoding, so as to involve sequence features into the output hidden states of PLM. We refer to our method as Self-augmented Sequentiality-sensitive Encoding (SSE for short). We carry out experiments on the benchmark SemEval datasets, including L-14, R-14, R-15 and R-16. Experimental results show that SSE yields substantial improvements, compared to BERT-based baselines. In particular, it is demonstrated that SSE is able to collaborate with other data augmentation approaches to produce more significant improvements, where the resultant ATE performance is up to 86.74%, 88.91%, 77.43% and 82.42% F1-scores (for L-14, R-14, R-15 and R-16 respectively).

Evaluating Large Language Models for Arabic Sentiment Analysis: A Comparative Study Using Retrieval-Augmented Generation

Sentiment Analysis (SA) is a crucial task in natural language processing. There are numerous studies devoted to this field specially with emerging use of encoder transformers and generative Large Language Models (LLMs). Transformer based models like BERT are often used for sentiment analysis. However, this article examines the performance of the latest generative generative LLMs in Arabic Sentiment Analysis (ASA) using Retrieval-Augmented Generation (RAG) architecture. We evaluated these models using the ASAD, ArSarcasm-v2, and SemEval datasets. Our experimental studies revealed challenges due to dataset imbalances and misclassified neutral labels, which impacted the effectiveness of fine-tuning. By removing the neutral class, significant improvements in model performance were observed across all datasets, with F1-scores increasing by 17%, 18%, and 18% on ASAD, ArSarcasm-v2, and SemEval, respectively.

A fine-grained deep learning model using embedded-CNN with BiLSTM for exploiting product sentiments

As technology advances, Facebook, Twitter, and microblogging sites have become the most effective platforms for communication and information exchange. Through these forums, people can share their views and experiences. These platforms enable discussion about a certain product that can be a valuable resource used to inform any decision-making process. For such studies, the majority of advanced-level researchers employed deep learning and machine learning models in conjunction with natural language processing (NLP). In recent years, the use of pre-trained models, such as Glove and BERT, in aspect-based sentiment analysis (ABSA) has increased. In ABSA, the auxiliary information is required to distinguish each aspect of this fine-grained task. However, BERT’s input format is restricted to a collection of words that cannot include more context knowledge. To address this problem, a BiLSTM and embedded CNN-based deep learning model has been presented for sentiment analysis at the aspect level. Initially, datasets were compiled from several sources. Then, an auxiliary feature was extracted using standard NLP. The auxiliary features were further refined and transformed into feature vectors based on the proposed embedded CNN model. Finally, a BiLSTM-based classification has been performed for sentiment classification. The experimental evaluation demonstrated that the performance of the suggested technique achieves on the SemEval dataset in terms of F1 score and accuracy by 81.7 and 83.3 percentage points, respectively, and on the product review dataset by 80.8 and 83.1 percentage points, respectively.

An Ensemble Model for Stance Detection in Social Media Texts

The aim of this paper is to develop a model to classify the stance expressed in social media texts. More specifically, the work presented focuses on tweets. In stance detection (SD) tasks, the objective is to identify the stance of a person towards a target of interest. In this paper, a model for SD is established and its variations are evaluated using different classifiers. The single models differ based on the pre-processing and the combination of features. To reduce the dimensionality of the feature space, analysis of variance (ANOVA) test is used. Then, two classifiers are employed as base learners including Random Forests (RF) and Support Vector Machines (SVM). Experimental analyses are conducted on SemEval dataset that is used as a benchmark for SD. Finally, the base learners that resulted from different design alternatives, are combined into three ensemble models. Experimental results show the significance of the used features and the effectiveness of a manually built dictionary that is used in the pre-processing stage. Moreover, the proposed ensembles outperform the state-of-the-art models in the overall test score, which suggests that ensemble learning is the best tool for effective SD in tweets.

A reinforced active learning approach for optimal sampling in aspect term extraction for sentiment analysis

Aspect level sentiment analysis is a fine grained task in sentiment analysis which identifies the product features from an opinionated piece of text and maps the sentiment towards each of them. Supervised ML algorithms have reported comparatively higher performance on aspect level sentiment analysis but at the cost of substantial qualitative labelled data. Data labelling for such fine grained tasks also demand domain knowledge and expertise. Hence a mechanism to extract a minimal informative subset which is almost representative of the entire data would be a breakthrough in bringing down the annotation costs to a large extent. The proposed methodology puts forward an active learning based sampling strategy for aspect term extraction, a subtask in aspect level sentiment analysis which identifies the product features. The sampling strategy is automated by reinforcement learning which extracts an optimal sample from the entire unlabelled training data and hence optimizes data annotation by reducing the time and effort linked to the labelling process. This work is of high importance in a data driven era where companies invest a lot in collecting and annotating huge volumes of data. The model has been experimented across the laptop and restaurant domains of SemEval (2014–2016) datasets. The experiments proved that a considerable reduction of the training data size is achieved across different datasets. The model trained on the data extracted by the proposed reinforced active learning model beats random sampling by 9 to 17 points when evaluated on the F-measure of the extracted aspect terms and is almost on par with the model trained on the entire training data by utilising hardly 9 to 13% of the entire training data across the datasets experimented.

A New Model for Emotions Analysis in Social Network Text Using Ensemble Learning and Deep learning

Recently, emotion analysis has become widely used. Therefore, increasing the accuracy of existing methods has become a challenge for researchers. The proposed method in this paper is a hybrid model to improve the accuracy of emotion analysis; Which uses a combination of convolutional neural network and ensemble learning. In the proposed method, after receiving the dataset, the data is pre-processed and converted into process able samples. Then the new dataset is split into two categories of training and test. The proposed model is a structure for machine learning in the form of ensemble learning. It contains blocks consisting of a combination of convolutional networks and basic classification algorithms. In each convolutional network, the base classification algorithms replace the fully connected layer. Evaluate the proposed method, in IMDB, PL04 and SemEval dataset with accuracy, precision, recall and F1 criteria, shows that, on average, for all three datasets, the precision of polarity detection is 90%, the recall of polarity detection is 93%, the F1 of polarity detection is 91% and finally the accuracy of polarity detection is 92%.

Aspect Category Detection

The major subtask of Sentiment analysis based on aspects (ABSA) is the category of aspectdetection (ACD). Due to the subjectivity inherent in categorizing, as well as the occurrence of overlapping classes, it is a difficult challenge to solve. Rule-based techniques, as well as other machine learning approaches, have been used to tackle ACD, and a majority of them are statistical behavior. We employed an association rulebased method in this article. We developed a mixed principle strategy that incorporates both association rule mining and semantics associations to address the statistical limitations of association rules. We employed the concept of word-embed for semantic linkages. The experiments were carried out using the SemEval dataset, which is a standardized set of data for categorizing features industry. We discovered how semantic connections could help to enhance classification accuracy by complementing statistical associations. The proposed method outperforms several statistical methods.

Word Sense Disambiguation Using Clustered Sense Labels

Sequence labeling models for word sense disambiguation have proven highly effective when the sense vocabulary is compressed based on the thesaurus hierarchy. In this paper, we propose a method for compressing the sense vocabulary without using a thesaurus. For this, sense definitions in a dictionary are converted into sentence vectors and clustered into the compressed senses. First, the very large set of sense vectors is partitioned for less computational complexity, and then it is clustered hierarchically with awareness of homographs. The experiment was done on the English Senseval and Semeval datasets and the Korean Sejong sense annotated corpus. This process demonstrated that the performance greatly increased compared to that of the uncompressed sense model and is comparable to that of the thesaurus-based model.

An unsupervised method for word sense disambiguation

Word sense disambiguation (WSD) finds the actual meaning of a word according to its context. This paper presents a novel WSD method to find the correct sense of a word present in a sentence. The proposed method uses both the WordNet lexical dictionary and the Wikipedia corpus. Initially, we find all the probable senses of the target word using WordNet. For each of the words present in a sense, we calculate the collocation extraction score with the other words in the sentence. The collocation extraction score finds the probability of the occurrence of two words together in the Wikipedia corpus. The maximum collocation extraction score assigns the proper sense for that context of the sentence. Our method is not limited to the bi-grams that are made up of only two consecutive words. Our method can find the probability of having two words together in a sentence when other words separate these two words. To compare our WSD method with current knowledge-based unsupervised and supervised systems, we use different Senseval and SemEval datasets for doing WSD on English words. Finally, the experimental analysis illustrates the significance of the proposed approach over many baseline and current systems.

Integrate syntax information for target-oriented opinion words extraction with target-specific graph convolutional network

Target-oriented Opinion Words Extraction (TOWE) aims to identify opinion words toward a specific target given the sentence. Syntax structure, which contains dependency relationships among words, is a vital clue for this task. With the help of syntax structure as a constraint, the model could remove irrelevant words and focus on tokens that are relevant to the given target. Directly adapting existing syntactic-based methods faces the problem that these models do not explicitly learn target-centric representations. Another challenge is that prior works only learn fixed order dependency relations, while context words require syntactic information in different scales. To handle these issues, we propose Target-Specific Graph Convolutional Network (TS-GCN) to explicitly integrate dependency structure. The proposed method could build high-quality syntax-aware representations by propagating target information to syntactically related words via graph convolution. Furthermore, we design a memory-based module to dynamically learn multi-granularity syntactic knowledge and infuse local features. Experimental results demonstrate the effectiveness of our method, and we achieve state-of-the-art performances on four SemEval datasets.

Aspect-gated graph convolutional networks for aspect-based sentiment analysis

Aspect-based sentiment analysis aims to predict the sentiment polarity of each specific aspect term in a given sentence. However, the previous models ignore syntactical constraints and long-range sentiment dependencies and mistakenly identify irrelevant contextual words as clues for judging aspect sentiment. In addition, these models usually use aspect-independent encoders to encode sentences, which can lead to a lack of aspect information. In this paper, we propose an aspect-gated graph convolutional network (AGGCN), that includes a special aspect gate designed to guide the encoding of aspect-specific information from the outset and construct a graph convolution network on the sentence dependency tree to make full use of the syntactical information and sentiment dependencies. The experimental results on multiple SemEval datasets demonstrate the effectiveness of the proposed approach, and our model outperforms the strong baseline models.

Commonsense Knowledge Enhanced Memory Network for Stance Classification

Stance classification aims at identifying, in the text, the attitude toward the given targets as favorable, negative, or unrelated. In existing models for stance classification, only textual representation is leveraged, while commonsense knowledge is ignored. In order to better incorporate commonsense knowledge into stance classification, we propose a novel model named commonsense knowledge enhanced memory network, which jointly represents textual and commonsense knowledge representation of given target and text. The textual memory module in our model treats the textual representation as memory vectors, and uses attention mechanism to embody the important parts. For commonsense knowledge memory module, we jointly leverage the entity and relation embeddings learned by TransE model to take full advantage of constraints of the knowledge graph. Experimental results on the SemEval dataset show that the combination of the commonsense knowledge memory and textual memory can improve stance classification.

Improve relation extraction with dual attention-guided graph convolutional networks

To better learn the dependency relationship between nodes, we address the relationship extraction task by capturing rich contextual dependencies based on the attention mechanism, and using distributional reinforcement learning to generate optimal relation information representation. This method is called Dual Attention Graph Convolutional Network (DAGCN), to adaptively integrate local features with their global dependencies. Specifically, we append two types of attention modules on top of GCN, which model the semantic interdependencies in spatial and relational dimensions, respectively. The position attention module selectively aggregates the feature at each position by a weighted sum of the features at all positions of nodes internal features. Meanwhile, the relation attention module selectively emphasizes interdependent node relations by integrating associated features among all nodes. We sum the outputs of the two attention modules and use reinforcement learning to predict the classification of nodes relationship to further improve feature representation which contributes to more precise extraction results. The results on the TACRED and SemEval datasets show that the model can obtain more useful information for relational extraction tasks, and achieve better performances on various evaluation indexes.

Supervised attention for answer selection in community question answering

<span class="fontstyle0">Answer selection is an important task in Community Question Answering (CQA). In recent years, attention-based neural networks have been extensively studied in various natural language processing problems, including question answering. This paper explores </span><span class="fontstyle2">matchLSTM </span><span class="fontstyle0">for answer selection in CQA. A lexical gap in CQA is more challenging as questions and answers typical contain multiple sentences, irrelevant information, and noisy expressions. In our investigation, word-by-word attention in the original model does not work well on social question-answer pairs. We propose integrating supervised attention into </span><span class="fontstyle2">matchLSTM</span><span class="fontstyle0">. Specifically, we leverage lexical-semantic from external to guide the learning of attention weights for question-answer pairs. The proposed model learns more meaningful attention that allows performing better than the basic model. Our performance is among the top on SemEval datasets.</span> <br /><br />

Aspect category detection using statistical and semantic association

AbstractAspect category detection (ACD) is an important subtask of aspect‐based sentiment analysis (ABSA). It is a challenging problem due to subjectivity involved in categorization, as well as the existence of overlapping classes. Among various approaches that have been applied to ACD include rule‐based approaches along with other machine learning approaches, and most of them are statistical in nature. In this article, we have used an association rule‐based approach. To deal with the statistical limitation of association rules, we proposed a hybridized rule‐based approach that combines association rules with the semantic association. For semantic associations, we have used the notion of word‐embeddings. Experiments were performed on SemEval dataset, a standard benchmark dataset for aspect categorization in the restaurant domain. We observed that semantic associations can complement statistical association and improve the accuracy of classification. The proposed method performs better than several state‐of‐the‐art methods.

Distilling Knowledge from Well-Informed Soft Labels for Neural Relation Extraction

Extracting relations from plain text is an important task with wide application. Most existing methods formulate it as a supervised problem and utilize one-hot hard labels as the sole target in training, neglecting the rich semantic information among relations. In this paper, we aim to explore the supervision with soft labels in relation extraction, which makes it possible to integrate prior knowledge. Specifically, a bipartite graph is first devised to discover type constraints between entities and relations based on the entire corpus. Then, we combine such type constraints with neural networks to achieve a knowledgeable model. Furthermore, this model is regarded as teacher to generate well-informed soft labels and guide the optimization of a student network via knowledge distillation. Besides, a multi-aspect attention mechanism is introduced to help student mine latent information from text. In this way, the enhanced student inherits the dark knowledge (e.g., type constraints and relevance among relations) from teacher, and directly serves the testing scenarios without any extra constraints. We conduct extensive experiments on the TACRED and SemEval datasets, the experimental results justify the effectiveness of our approach.

Towards End-to-End Multilingual Question Answering

Multilingual question answering (MLQA) is a critical part of an accessible natural language interface. However, current solutions demonstrate performance far below that of monolingual systems. We believe that deep learning approaches are likely to improve performance in MLQA drastically. This work aims to discuss the current state-of-the-art and remaining challenges. We outline requirements and suggestions for practical parallel data collection and describe existing methods, benchmarks and datasets. We also demonstrate that a simple translation of texts can be inadequate in case of Arabic, English and German languages (on InsuranceQA and SemEval datasets), and thus more sophisticated models are required. We hope that our overview will re-ignite interest in multilingual question answering, especially with regard to neural approaches.

Pre-trained Language Embedding-based Contextual Summary and Multi-scale Transmission Network for Aspect Extraction

With the development of IOT and 5G technology, people’s demand for information acquisition is more inclined to accuracy, intelligence and timeliness. How to help designer obtain the real-time information of specific product reviews from the massive online consumers and upgrade the new design strategy has become a hot topic for research. In this paper, we define the problem as an aspect extraction task, and propose a novel deep learning model that comprises of three modules: pre-training language model embedding, multi-scale transmission network and contextual summary, which aims to provide an end-to-end solution without any additional supervision. To this end, we adopt BERT to overcome the disadvantage of traditional embedding methods, which cannot combine contextual information. Multi-scale transmission network is proposed to integrate the Bi-GRU and a group of CNN networks to extract sequential and local features of words respectively. Contextual summary is a tailor-made representation distilled from the input sentence, conditioned on each current word, and thus can assist aspect prediction. Experimental results over three benchmark SemEval datasets clearly illustrate that our model can achieve the state-of-the-art performance.

Improving aspect-based sentiment analysis via aligning aspect embedding

Aspect-Based Sentiment Analysis (ABSA) is a fine-grained sentiment analysis task, which aims to predict sentiment polarities of given aspects or target terms in text. ABSA contains two subtasks: Aspect-Category Sentiment Analysis (ACSA) and Aspect-Term Sentiment Analysis (ATSA). Aspect embeddings have been extensively used for representing aspect-categories on ACSA task. Based on our observations, existing aspect embeddings cannot properly represent the relation between aspect-categories and aspect-terms. To address this limitation, this paper presents a learning method which trains aspect embeddings according to the relation between aspect-categories and aspect-terms. According to the cosine measure metric we proposed in this paper, the limitation is successfully alleviated in the aspect embeddings which are trained by our method. The trained aspect embeddings can be used as initialization in existing models to solve ACSA task. We conduct experiments on SemEval datasets for ACSA task, and the results indicate that our pre-trained aspect embeddings are capable of improving the performance of sentiment analysis.