SemEval-2019 Task Research Articles

GENERIC APPROACH OF MEASURING TEXT SEMANTIC SIMILARITY

Text Semantic Similarity can be viewed as one of the challenging tasks as evident from current profound interest in NLP research community that has created achievable milestones through active participation in SemEval task series of the recent decade. Amidst these developments, it was realized that exploring text to compare its semantics largely depends on valid grammatical structures of sentences and sentence formulation types. In this paper, the computation of text semantic similarity is addressed by devising a novel set of generic similarity metrics based on both, word-sense of the phrases constituting the text as well as the grammatical layout and sequencing of these word-phrases forming text with sensible meaning. We have used the combination of word-sense and grammatical similarity metrics over benchmark sentential datasets. Having obtained highest value of Pearson’s correlation coefficient (0.89) with mean human similarity scores, when compared against equivalent scores obtained through closely competent structured approach models, plagiarism-detection classification task was revisited on well-known paragraph-phrased Rewrite corpus articulated by Clough and Stevenson (2011) using our model to provide generic utility perspective to these novel devised similarity metrics. Here also, nearly competent classification model performance (with accuracy 76.8%) encouraged authors to work in directions that are more promising where the performance can be enhanced by improving upon dependency (grammatical relations) component in order to raise the count of true-positives and false-negatives.

Arabic Biomedical Community Question Answering Based on Contextualized Embeddings

Community question answering has become increasingly important as they are practical for seeking and sharing information. Applying deep learning models often leads to good performance, but it requires an extensive amount of annotated data, a problem exacerbated for languages suffering a scarcity of resources. Contextualized language representation models have gained success due to promising results obtained on a wide array of downstream natural language processing tasks such as text classification, textual entailment, and paraphrase identification. This paper presents a novel approach by fine-tuning contextualized embeddings for a medical domain community question answering task. The authors propose an architecture combining two neural models powered by pre-trained contextual embeddings to learn a sentence representation and thereafter fine-tuned on the task to compute a score used for both ranking and classification. The experimental results on SemEval Task 3 CQA show that the model significantly outperforms the state-of-the-art models by almost 2% for the '16 edition and 1% for the '17 edition.

Sentiment Analysis Method of Network Text Based on Improved AT-BiGRU Model

In order to solve the problems existing in the current method of emotional analysis of network text, such as long training time, complex calculation, and large space cost, this paper proposes an Internet text sentiment analysis method based on the improved AT-BiGRU model. Firstly, the textblob package is imported to correct spelling errors before text preprocessing. Secondly, pad_sequences are used to fill in the input layer with a fixed length, the two-way gated recurrent network is used to extract information, and the attention mechanism is used to highlight the key information of the word vector. Finally, the GNU memory unit is transformed, and an improved BiGRU that can adapt to the recursive network structure is constructed. The proposed model is experimentally demonstrated on the SemEval-2014 Task 4 and SemEval-2017 Task 4 datasets. Experimental results show that the proposed model can effectively avoid the text sentiment analysis bias caused by spelling errors and prove the effectiveness of the improved AT-BiGRU model in terms of accuracy, loss rate, and iteration time.

Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis

Aspect-based sentiment analysis (ABSA) and Targeted ASBA (TABSA) allow finer-grained inferences about sentiment to be drawn from the same text, depending on context. For example, a given text can have different targets (e.g., neighborhoods) and different aspects (e.g., price or safety), with different sentiment associated with each target-aspect pair. In this paper, we investigate whether adding context to self-attention models improves performance on (T)ABSA. We propose two variants of Context-Guided BERT (CG-BERT) that learn to distribute attention under different contexts. We first adapt a context-aware Transformer to produce a CG-BERT that uses context-guided softmax-attention. Next, we propose an improved Quasi-Attention CG-BERT model that learns a compositional attention that supports subtractive attention. We train both models with pretrained BERT on two (T)ABSA datasets: SentiHood and SemEval-2014 (Task 4). Both models achieve new state-of-the-art results with our QACG-BERT model having the best performance. Furthermore, we provide analyses of the impact of context in the our proposed models. Our work provides more evidence for the utility of adding context-dependencies to pretrained self-attention-based language models for context-based natural language tasks.

Topic-level sentiment analysis of social media data using deep learning

Due to the inception of Web 2.0 and freedom to facilitate the dissemination of information, sharing views, expressing opinions with regards to current world level events, services, products, etc. social media platforms have been mainly contributing to user-generated content. Such social media data consist of various themes discussed online and are associated with sentiments of the users. To catch up with the speed of streaming data at which it generates on social media platforms, it is crucial to detect the topics being discussed on social media platforms and analyze the sentiments of users towards those topics in an online manner to make timely decisions. Motivated by the same, this paper proposes a deep learning based topic-level sentiment analysis model. The novelty of the proposed approach is that it works at the sentence level to extract the topic using online latent semantic indexing with regularization constraint and then applies topic-level attention mechanism in long short-term memory network to perform sentiment analysis. The proposed model is unique in the sense that it supports scalable and dynamic topic modeling over streaming short text data and performs sentiment analysis at topic-level. For SemEval-2017 Task 4 Subtask B dataset as a case of in-domain topic-level sentiment analysis, average recall of 0.879 has been achieved, whereas, for out-of-domain data, average recall of 0.846, 0.824 and 0.794 has been achieved for newly developed datasets collected under the hashtags #ethereum, #bitcoin and #facebook from Twitter. To assess the performance of the model for scalability, we analyzed the model in terms of average time in milliseconds for creation of feature vectors, throughput in terms of topics detected per second and average response time in seconds to handle the sentiment analysis queries. The experimental results are significant enough to enable large scale topic modeling over streaming data and perform topic-level sentiment analysis.

A Multi-Layer Network for Aspect-Based Cross-Lingual Sentiment Classification

In the recent era, the advancement of communication technologies provides a valuable interaction source between people of different regions. Nowadays, many organizations adopt the latest approaches, i.e., sentiment analysis and aspect-oriented sentiment classification, to evaluate user reviews to improve the quality of their products. The processing of multi-lingual user reviews is a key challenge in Natural Language Processing (NLP). This paper proposes a multi-layer network with divided attention to perform aspect-based sentiment classification for cross-lingual data. It extracts the Part-of-Speech (POS) tagging information of the given reviews, preprocesses them, and converts them into tokens. Furthermore, bi-lingual dictionaries are leveraged to map the converted tokens from one language to another. Given the preprocessed and mapped reviews, vectors are generated by leveraging the multi-lingual BERT and passed to the proposed deep learning classifier. The 10351 restaurant reviews from SemEval-2016 Task 5 dataset are exploited for the prediction of aspect-based sentiment. The results of cross-lingual validation suggest that the proposed approach significantly outperforms the state-of-the-art approaches and improves the precision, recall, and F1 by more than 23%, 20%, and 22%, respectively.

Combining Context-Aware Embeddings and an Attentional Deep Learning Model for Arabic Affect Analysis on Twitter

Affect analysis has recently attracted a great deal of attention due to the rapid development of online social platforms (i.e., Twitter, Facebook). Affect analysis is a part of a broader area of affective computing that aims to detect and grasp human emotions or affects within a piece of writing. Context awareness is very relevant for identifying human emotions and affects behind a piece of text. Capturing the context of a piece of text is often perceived as a challenge. In addition to the own unique features of tweets (shortness, noisiness, short length, etc.), the Arabic language is characterized by its agglutination and morphological richness. In this paper, we address the problem of Arabic affect detection (multilabel emotion classification) by combining the transformer-based model for Arabic language understanding AraBERT and an attention-based LSTM-BiLSTM deep model. AraBERT generates the contextualized embedding, and the attention-based LSTM-BiLSTM determines the label-emotion of tweets by extracting both past and future contexts considering temporal information flow in both directions. Additionally, the attention mechanism is applied to the output of LSTM-BiLSTM to emphasize different words. Our proposed approach was evaluated using the reference dataset of SemEval-2018 Task 1 (Affect in Tweets). The comprehensive results show that the proposed approach outperforms eight current state-of-the-art and baseline methods, and it achieves significant accuracy (53.82%) compared to 1st place in SemEval2018-Task1: (Affect in Tweets) competition. In addition, our proposed model outperforms the best recently reported model in the literature, with an enhancement of 2.62% in accuracy.

Residual Fusion Network Based Attention for Aspect Level Sentiment Analysis

Aspect level sentiment analysis is a more fine-grain task compared to document level and sentence level. The aim of the task is to identify the sentiment polarity expressed to the given aspect in the text. In this paper, we propose a residual aspect fusion network with attention for the aspect level sentiment classification. In this network, bidirectional GRU is used to collect semantic information of both sentence words and aspect words, and then attention mechanism is applied to construct aspect vectors for each word in the sentence. In the residual fusion module, the model fuses the aspect information and sentence information together by residual connection structure, in which position weight is applied to utilize the position information of aspect words. At the end of the model, Transformer encode layer is used to extract global feature and fully connected layers are used as the classifier. We train and test the network on SemEval2014 task 4 (Restaurant and Laptop) and Twitter datasets, and the experiments demonstrate that our model performs better than previous methods.

A novel approach to stance detection in social media tweets by fusing ranked lists and sentiments

Stance detection is a relatively new concept in data mining that aims to assign a stance label (favor, against, or none) to a social media post towards a specific pre-determined target. These targets may not be referred to in the post, and may not be the target of opinion in the post. In this paper, we propose a novel enhanced method for identifying the writer’s stance of a given tweet. This comprises a three-phase process for stance detection: (a) tweets preprocessing; here we clean and normalize tweets (e.g., remove stop-words) to generate words and stems lists, (b) features generation; in this step, we create and fuse two dictionaries for generating features vector, and lastly (c) classification; all the instances of the features are classified based on the list of targets. Our innovative feature selection proposes fusion of two ranked lists (top-k) of term frequency-inverse document frequency (tf-idf) scores and the sentiment information. We evaluate our method using six different classifiers: K nearest neighbor (K-NN), discernibility-based K-NN, weighted K-NN, class-based K-NN, exemplar-based K-NN, and Support Vector Machines. Furthermore, we investigate the use of Principal Component Analysis and study its effect on performance. The model is evaluated on the benchmark dataset (SemEval-2016 task 6), and the results significance is determined using t-test. We achieve our best performance of macro F-score (averaged across all topics) of 76.45% using the weighted K-NN classifier. This tops the current state-of-the-art score of 74.44% on the same dataset.

Convolutional multi-head self-attention on memory for aspect sentiment classification

This paper presents a method for aspect based sentiment classification tasks, named convolutional multi-head self-attention memory network (CMA-MemNet). This is an improved model based on memory networks, and makes it possible to extract more rich and complex semantic information from sequences and aspects. In order to fix the memory network's inability to capture context-related information on a word-level, we propose utilizing convolution to capture n-gram grammatical information. We use multi-head self-attention to make up for the problem where the memory network ignores the semantic information of the sequence itself. Meanwhile, unlike most recurrent neural network (RNN) long short term memory (LSTM), gated recurrent unit (GRU) models, we retain the parallelism of the network. We experiment on the open datasets SemEval-2014 Task 4 and SemEval-2016 Task 6. Compared with some popular baseline methods, our model performs excellently.

DCMN+: Dual Co-Matching Network for Multi-Choice Reading Comprehension

Multi-choice reading comprehension is a challenging task to select an answer from a set of candidate options when given passage and question. Previous approaches usually only calculate question-aware passage representation and ignore passage-aware question representation when modeling the relationship between passage and question, which cannot effectively capture the relationship between passage and question. In this work, we propose dual co-matching network (DCMN) which models the relationship among passage, question and answer options bidirectionally. Besides, inspired by how humans solve multi-choice questions, we integrate two reading strategies into our model: (i) passage sentence selection that finds the most salient supporting sentences to answer the question, (ii) answer option interaction that encodes the comparison information between answer options. DCMN equipped with the two strategies (DCMN+) obtains state-of-the-art results on five multi-choice reading comprehension datasets from different domains: RACE, SemEval-2018 Task 11, ROCStories, COIN, MCTest.

Integer programming ensemble of temporal relations classifiers

The extraction of temporal events from text and the classification of temporal relations among both temporal events and time expressions are major challenges for the interface of data mining and natural language processing. We present an ensemble method, which reconciles the outputs of multiple heterogenous classifiers of temporal expressions. We use integer programming, a constrained optimisation technique, to improve on the best result of any individual classifier by choosing consistent temporal relations from among those recommended by multiple classifiers. Our ensemble method is conceptually simple and empirically powerful. It allows us to encode knowledge about the structure of valid temporal expressions as a set of constraints. It obtains new state-of-the-art results on two recent natural language processing challenges, SemEval-2013 TempEval-3 (Temporal Annotation) and SemEval-2016 Task 12 (Clinical TempEval), with F1 scores of 0.3915 and 0.595 respectively.

Aspect Based Sentimental Analysis of Hotel Reviews: A Comparative Study

The increasing use of the internet enables users to share their opinion about what they like and dislike regarding products and services. For efficient decision making, there is a need to analyze these reviews. Sentiment analysis or opinion mining is commonly used to detect polarity (positive or negative) of reviews. But, it does not show the aspect or orientation of the text. In this study, state-of-art approaches based on supervised machine learning employed to perform three tasks on the dataset provided by SemEval. Tasks A and B are related to predicting the aspect of the restaurant’s reviews, whereas task C shows their polarity. Additionally, this study aims to compare the performance of two feature engineering techniques and five machine learning algorithms to evaluate their performance on a publicly available dataset named SemEval-2015 Task 12. The experimental results showed that the word2vec features when used with the support vector machine algorithm outperformed by giving 76%, 72% and 79% off overall accuracies for Task A, Task B, and Task C respectively. Our comparative study holds practical significance and can be used as a baseline study in the domain of aspect-based sentiment analysis.

A Deep Learning-Based Approach for Multi-Label Emotion Classification in Tweets

Currently, people use online social media such as Twitter or Facebook to share their emotions and thoughts. Detecting and analyzing the emotions expressed in social media content benefits many applications in commerce, public health, social welfare, etc. Most previous work on sentiment and emotion analysis has only focused on single-label classification and ignored the co-existence of multiple emotion labels in one instance. This paper describes the development of a novel deep learning-based system that addresses the multiple emotion classification problem in Twitter. We propose a novel method to transform it to a binary classification problem and exploit a deep learning approach to solve the transformed problem. Our system outperforms the state-of-the-art systems, achieving an accuracy score of 0.59 on the challenging SemEval2018 Task 1:E-cmulti-label emotion classification problem.

A Single Attention-Based Combination of CNN and RNN for Relation Classification

As a vital task in natural language processing, relation classification aims to identify relation types between entities from texts. In this paper, we propose a novel Att-RCNN model to extract text features and classify relations by combining recurrent neural network (RNN) and convolutional neural network (CNN). This network structure utilizes RNN to extract higher level contextual representations of words and CNN to obtain sentence features for the relation classification task. In addition to this network structure, both word-level and sentence-level attention mechanisms are employed in Att-RCNN to strengthen critical words and features to promote the model performance. Moreover, we conduct experiments on four distinct datasets: SemEval-2010 task 8, SemEval-2018 task 7 (two subtask datasets), and KBP37 dataset. Compared with the previous public models, Att-RCNN has the overall best performance and achieves the highest $F_{1}$ score, especially on the KBP37 dataset.

CONVOLUTIONAL NEURAL NETWORK MULTI-EMOTION CLASSIFIERS

The natural languages are universal and flexible but cannot exist without ambiguity. Having more than one attitude and meaning in the same phrase context is the main cause for word or phrase ambiguity. Most previous work on emotion analysis has only coverage single-label classification and neglect the presence of multiple emotion labels in one instance. This paper presents multi emotion classification in Twitter based on Convolutional Neural Networks (CNN). The applied features are emotion lexicons, word embeddings, and frequency distribution. The proposed networks performance is evaluated to state-of-the-art classification algorithms, achieving hamming score ranges from 0.46 to 0.52 on the challenging SemEval2018 Task E-c.

Event Time Extraction with a Decision Tree of Neural Classifiers

Extracting the information from text when an event happened is challenging. Documents do not only report on current events, but also on past events as well as on future events. Often, the relevant time information for an event is scattered across the document. In this paper we present a novel method to automatically anchor events in time. To our knowledge it is the first approach that takes temporal information from the complete document into account. We created a decision tree that applies neural network based classifiers at its nodes. We use this tree to incrementally infer, in a stepwise manner, at which time frame an event happened. We evaluate the approach on the TimeBank-EventTime Corpus (Reimers et al., 2016) achieving an accuracy of 42.0% compared to an inter-annotator agreement (IAA) of 56.7%. For events that span over a single day we observe an accuracy improvement of 33.1 points compared to the state-of-the-art CAEVO system (Chambers et al., 2014). Without retraining, we apply this model to the SemEval-2015 Task 4 on automatic timeline generation and achieve an improvement of 4.01 points F1-score compared to the state-of-the-art. Our code is publically available.

Heterographic pun identification model based on multi-dimensional semantic relationships

Identifying heterographic puns is an important branch of humor research, which has gradually developed into a new research area. This paper presents a heterographic pun identification mechanism based on feature sets in four dimensions, namely, semantic transparency, semantic relevance, phonetic expansibility, and syntax feature sets. The semantic transparency feature sets consist of the lexical item statistics and the character length; the syntax feature sets include names, capitalization, tense, part of speech, and location. Nine features of the above four dimensions are added to a binary decision tree to generate a threshold and complete a pun identification with the help of K-means clustering. Using the corpus of the SemEval2017 Task 7, the proposed method achieves satisfactory results, and its F1 value outscores the top one out of all participating teams. The experiment outlined in this paper proves that the taxonomic approach of the binary decision tree algorithm based on four dimensions is effective in identifying heterographic puns. The phonetic expansibility and the syntax feature sets are particularly effective among all other dimensions, which is consistent with our presumption that the phonetic feature plays a bigger role in identifying heterographic puns.

Understanding Citizen Issues through Reviews: A Step towards Data Informed Planning in Smart Cities

Governments these days are demanding better Smart City technologies in order to connect with citizens and understand their demands. For such governments, much needed information exists on social media where members belonging to diverse groups share different interests, post statuses, review and comment on various topics. Aspect extraction from this data can provide a thorough understanding of citizens’ behaviors and choices. Also, categorization of these aspects can better summarize societal concerns regarding political, economic, religious and social issues. Aspect category detection (ACD) from people reviews is one of the major tasks of aspect-based sentiment analysis (ABSA). The success of ABSA is mainly defined by the inexpensive and accurate machine-processable representation of the raw input sentences. Previous approaches rely on cumbersome feature extraction procedures from sentences, which adds its own complexity and inaccuracy in performing ACD tasks. In this paper, we propose an inexpensive and simple method to obtain the most suitable representation of a sentence-vector through different algebraic combinations of a sentence’s word vectors, which will act as an input to any machine learning classifier. We have tested our technique on the restaurant review data provided in SemEval-2015 and SemEval-2016. SemEval is a series of global challenges to evaluate the effectiveness of disambiguation of word sense. Our results showed the highest F1-scores of 76.40% in SemEval-2016 Task 5, and 94.99% in SemEval-2015 Task 12.

A Neural Transition-Based Approach for Semantic Dependency Graph Parsing

Semantic dependency graph has been recently proposed as an extension of tree-structured syntactic or semantic representation for natural language sentences. It particularly features the structural property of multi-head, which allows nodes to have multiple heads, resulting in a directed acyclic graph(DAG) parsing problem. Yet most statistical parsers focused exclusively on shallow bi-lexical tree structures, DAG parsing remains under-explored. In this paper, we propose a neural transition-based parser, using a variant of list-based arc-eager transition algorithm for dependency graph parsing. Particularly, two non-trivial improvements are proposed for representing the key components of the transition system, to better capture the semantics of segments and internal sub-graph structures. We test our parser on the SemEval-2016 Task 9 dataset (Chinese) and the SemEval-2015 Task 18 dataset (English). On both benchmark datasets, we obtain superior or comparable results to the best performing systems. Our parser can be further improved with a simple ensemble mechanism, resulting in the state-of-the-art performance.