Stance Classification Research Articles

GPT-4 as an X data annotator: Unraveling its performance on a stance classification task.

Data annotation in NLP is a costly and time-consuming task, traditionally handled by human experts who require extensive training to enhance the task-related background knowledge. Besides, labeling social media texts is particularly challenging due to their brevity, informality, creativity, and varying human perceptions regarding the sociocultural context of the world. With the emergence of GPT models and their proficiency in various NLP tasks, this study aims to establish a performance baseline for GPT-4 as a social media text annotator. To achieve this, we employ our own dataset of tweets, expertly labeled for stance detection with full inter-rater agreement among three annotators. We experiment with three techniques: Zero-shot, Few-shot, and Zero-shot with Chain-of-Thoughts to create prompts for the labeling task. We utilize four training sets constructed with different label sets, including human labels, to fine-tune transformer-based large language models and various combinations of traditional machine learning models with embeddings for stance classification. Finally, all fine-tuned models undergo evaluation using a common testing set with human-generated labels. We use the results from models trained on human labels as the benchmark to assess GPT-4's potential as an annotator across the three prompting techniques. Based on the experimental findings, GPT-4 achieves comparable results through the Few-shot and Zero-shot Chain-of-Thoughts prompting methods. However, none of these labeling techniques surpass the top three models fine-tuned on human labels. Moreover, we introduce the Zero-shot Chain-of-Thoughts as an effective strategy for aspect-based social media text labeling, which performs better than the standard Zero-shot and yields results similar to the high-performing yet expensive Few-shot approach.

Zero-shot stance detection based on multi-expert collaboration

Zero-shot stance detection is pivotal for autonomously discerning user stances on novel emerging topics. This task hinges on effective feature alignment transfer from known to unseen targets. To address this, we introduce a zero-shot stance detection framework utilizing multi-expert cooperative learning. This framework comprises two core components: a multi-expert feature extraction module and a gating mechanism for stance feature selection. Our approach involves a unique learning strategy tailored to decompose complex semantic features. This strategy harnesses the expertise of multiple specialists to unravel and learn diverse, intrinsic textual features, enhancing transferability. Furthermore, we employ a gating-based mechanism to selectively filter and fuse these intricate features, optimizing them for stance classification. Extensive experiments on standard benchmark datasets demonstrate that our model significantly surpasses existing baseline models in performance.

A Deep Learning Ensemble Framework for Fake News Stance Detection and Classification

A Deep Learning Ensemble Framework for Fake News Stance Detection and Classification

DSMM: A dual stance-aware multi-task model for rumour veracity on social networks

Rumour veracity on social networks aims to determine the authenticity of a claim based on the conversation graph. Current works typically employ graph neural networks (GNNs) to model the conversation graph and leverage the users’ stance classification on the claim (referred to as global stance) to enhance rumour veracity with multi-task learning methods. However, due to the over-smoothing problem, the layers of GNNs are usually very shallow. In Consequence, the graph nodes with limited receptive fields cannot perceive the semantic information of the claim node. This results in GNNs being sub-optimal for global stance learning. To address this issue, we present a novel dual stance-aware multi-task model, DSMM, to improve the rumour veracity results. Firstly, we design a position-aware semantic compensation method, which relies on the relative topological positions between the claim node and comment nodes to compensate graph nodes with different degrees of claim semantics, thus enhancing the awareness of graph nodes for claim semantics. In addition, we propose using local stance, the attitude tendency of comment nodes towards their predecessor nodes, to model local interactions of the conversation graph for the first time. Since the local stance is a fine-grained interaction representation of the conversation graph, it can complement the global stance and enhance the model’s understanding for conversation graphs. Further, we introduce the directed attention network to simulate the directedness of local stances and differentiate the importance of different local stances simultaneously. Finally, we use the cross-attention technique to fuse information from the above two stance classifications into the rumour veracity module to enhance model performance further. Experimental results on two publicly available competition datasets demonstrate the effectiveness of our approach.

Stance Classification for Fake News Detection with Machine Learning

Stance Classification for Fake News Detection with Machine Learning

Understanding stance classification of BERT models: an attention-based framework

Understanding stance classification of BERT models: an attention-based framework

An Explainable Fake News Analysis Method with Stance Information

The high level of technological development has enabled fake news to spread faster than real news in cyberspace, leading to significant impacts on the balance and sustainability of current and future social systems. At present, collecting fake news data and using artificial intelligence to detect fake news have an important impact on building a more sustainable and resilient society. Existing methods for detecting fake news have two main limitations: they focus only on the classification of news authenticity, neglecting the semantics between stance information and news authenticity. No cognitive-related information is involved, and there are not enough data on stance classification and news true-false classification for the study. Therefore, we propose a fake news analysis method based on stance information for explainable fake news detection. To make better use of news data, we construct a fake news dataset built on cognitive information. The dataset primarily consists of stance labels, along with true-false labels. We also introduce stance information to further improve news falsity analysis. To better explain the relationship between fake news and stance, we use propensity score matching for causal inference to calculate the correlation between stance information and true-false classification. The experiment result shows that the propensity score matching for causal inference yielded a negative correlation between stance consistency and fake news classification.

Meta-learning with topic-agnostic representations for zero-shot stance detection

Zero-shot stance detection (ZSSD) aims to classify stances on unseen topic data with the ability of a model to generalize in the range of topics in the real world. The key point of ZSSD is to prevent the model from overfitting on seen topic data and show robust stance recognition performance on unseen topic samples. In this paper, we propose a new ZSSD framework to generalize stance detection performance by alleviating seen data information from the encoder and focusing on improving stance classification ability in zero-shot conditions. The proposed framework involves two learning stages contributing to ZSSD: topic-agnostic text encoder learning and zero-shot meta-learning. Our framework achieves notable improvements on the three benchmark zero-shot stance detection datasets and zero-shot aspect target sentiment classification dataset showing the effectiveness of our method in the zero-shot settings.

Performing Stance Classification and Bot Detection on the Indian Farmers’ Protest – A Study to Unveil Hidden Perspectives.

The presence of illegal, harmful content, rumors, misinformation, and Twitter bots has consistently brought the social media platforms such as Twitter into the spotlight. Therefore, it is advisable to exercise caution when analyzing tweets. To establish the credibility of any patterns and findings derived from tweets, it is essential to thoroughly investigate the source and authenticity of the tweets in question. This paper advances in this direction by introducing a novel approach involving bot detection and a comparative analysis of human and botgenerated tweets related to the farmers’ protest. A framework for knowledge differentiation is deployed to accomplish this goal. The framework unearths the global perspectives of people about Indian farmers’ protests, in the form of stances, the results of which serve as nuggets of knowledge derived at the lower level of abstraction. Unexpected results of stance detection motivated the study of bot detection in each tweet of each stance. Knowledge discovered by bot detection and characterization studies was thus built over stance detection and yielded higher-order knowledge nuggets, which identified the widespread presence of bots in tweets both for and against the protest, thus establishing the misuse of social media platforms like Twitter to influence and control the narrative of the social events that significantly impact people’s lives. Characterization of issues being tweeted by humans vs. bots in favor of and against farmers’ protests was accomplished by conducting a comparative analysis of Ngrams in each category. Vocabulary analysis established that texts tweeted by bots mimicked the vocabulary pattern of the tweets by human users. Research inferences such as these can be invaluable for policy makers, enabling them to gain a macro-level understanding of the situations on the ground level and leverage such information for making policy decisions, in order to be prepared to handle similar situations in the future.

Investigating Opinions on Public Policies in Digital Media: Setting up a Supervised Machine Learning Tool for Stance Classification

ABSTRACT Supervised machine learning (SML) provides us with tools to efficiently scrutinize large corpora of communication texts. Yet, setting up such a tool involves plenty of decisions starting with the data needed for training, the selection of an algorithm, and the details of model training. We aim at establishing a firm link between communication research tasks and the corresponding state-of-the-art in natural language processing research by systematically comparing the performance of different automatic text analysis approaches. We do this for a challenging task – stance detection of opinions on policy measures to tackle the COVID-19 pandemic in Germany voiced on Twitter. Our results add evidence that pre-trained language models such as BERT outperform feature-based and other neural network approaches. Yet, the gains one can achieve differ greatly depending on the specific merits of pre-training (i.e., use of different language models). Adding to the robustness of our conclusions, we run a generalizability check with a different use case in terms of language and topic. Additionally, we illustrate how the amount and quality of training data affect model performance pointing to potential compensation effects. Based on our results, we derive important practical recommendations for setting up such SML tools to study communication texts.

Replicable semi-supervised approaches to state-of-the-art stance detection of tweets

Stance is defined as the expression of a speaker's standpoint towards a given target or entity. To date, the most reliable method for measuring stance is opinion surveys. However, people's increased reliance on social media makes these online platforms an essential source of complementary information about public opinion. Our study contributes to the discussion surrounding replicable methods through which to conduct reliable stance detection by establishing a rule-based model, which we replicated for several targets independently. To test our model, we relied on a widely used dataset of annotated tweets - the SemEval Task 6A dataset, which contains 5 targets with 4,163 manually labelled tweets. We relied on “off-the-shelf” sentiment lexica to expand the scope of our custom dictionaries, while also integrating linguistic markers and using word-pairs dependency information to conduct stance classification. While positive and negative evaluative words are the clearest markers of expression of stance, we demonstrate the added value of linguistic markers to identify the direction of the stance more precisely. Our model achieves an average classification accuracy of 75% (ranging from 67% to 89% across targets). This study is concluded by discussing practical implications and outlooks for future research, while highlighting that each target poses specific challenges to stance detection.

Stance Classification of Social Media Texts for Under-Resourced Scenarios in Social Sciences

In this work, we explore the performance of supervised stance classification methods for social media texts in under-resourced languages and using limited amounts of labeled data. In particular, we focus specifically on the possibilities and limitations of the application of classic machine learning versus deep learning in social sciences. To achieve this goal, we use a training dataset of 5.7K messages posted on Flashback Forum, a Swedish discussion platform, further supplemented with the previously published ABSAbank-Imm annotated dataset, and evaluate the performance of various model parameters and configurations to achieve the best training results given the character of the data. Our experiments indicate that classic machine learning models achieve results that are on par or even outperform those of neural networks and, thus, could be given priority when considering machine learning approaches for similar knowledge domains, tasks, and data. At the same time, the modern pre-trained language models provide useful and convenient pipelines for obtaining vectorized data representations that can be combined with classic machine learning algorithms. We discuss the implications of their use in such scenarios and outline the directions for further research.

AN EFFECTIVE IDENTIFICATION OF TRUTHFUL INFORMATION USING POLYMERIZATION SENTIMENTAL MODLEL

Despite the increasing use of social media platforms for information and news gathering, its immoderate nature will create peoples like keyboard warriors, rumormongers etc. This often leads to the emergence and spreading of rumors, i.e. pieces of information that are unverified at the time of posting. At the same time, the openness of social media platforms provides opportunities to study how users share and discuss rumors, and to explore how natural language processing and data mining techniques may be used to find ways of determining their veracity. In this survey, we introduce the Big Data technology and discuss two types of rumors that circulate on social media; long-standing rumors that circulate for long periods of time, and newlyemerging rumors spawned during fast-paced events such as breaking news, where reports are released piecemeal and often with an unverified status in their early stages. We provide an overview of datasets into social media rumors with the ultimate goal of developing a rumor classification system that consists of four components: rumor detection, rumor tracking, rumor stance classification and rumor veracity classification. We delve into the approaches presented in the scientific literature for the development of each of these four components. We summarize the efforts and achievements so far towards the development of rumor classification systems and conclude with suggestions for avenues for future research in social media mining for detection and resolution of rumors

Adaptive cost-sensitive stance classification model for rumor detection in social networks

As online social networks are experiencing extreme popularity growth, determining the veracity of online statements denoted by rumors automatically as earliest as possible is essential to prevent the harmful effects of propagating misinformation. Early detection of rumors is facilitated by considering the wisdom of the crowd through analyzing different attitudes expressed toward a rumor (i.e., users’ stances). Stance detection is an imbalanced problem as the querying and denying stances against a given rumor are significantly less than supportive and commenting stances. However, the success of stance-based rumor detection significantly depends on the efficient detection of “query” and “deny” classes. The imbalance problem has led the previous stance classifier models to bias toward the majority classes and ignore the minority ones. Consequently, the stance and subsequently rumor classifiers have been faced with the problem of low performance. This paper proposes a novel adaptive cost-sensitive loss function for learning imbalanced stance data using deep neural networks, which improves the performance of stance classifiers in rare classes. The proposed loss function is a cost-sensitive form of cross-entropy loss. In contrast to most of the existing cost-sensitive deep neural network models, the utilized cost matrix is not manually set but adaptively tuned during the learning process. Hence, the contributions of the proposed method are both in the formulation of the loss function and the algorithm for calculating adaptive costs. The experimental results of applying the proposed algorithm to stance classification of real Twitter and Reddit data demonstrate its capability in detecting rare classes while improving the overall performance. The proposed method improves the mean F-score of rare classes by about 13% in RumorEval 2017 dataset and about 20% in RumorEval 2019 dataset.

Is my stance the same as your stance? A cross validation study of stance detection datasets

Stance detection identifies a person’s evaluation of a subject, and is a crucial component for many downstream applications. In application, stance detection requires training a machine learning model on an annotated dataset and applying the model on another to predict stances of text snippets. This cross-dataset model generalization poses three central questions, which we investigate using stance classification models on 7 publicly available English Twitter datasets ranging from 297 to 48,284 instances. (1) Are stance classification models generalizable across datasets? We construct a single dataset model to train/test dataset-against-dataset, finding models do not generalize well (avg F1=0.33). (2) Can we improve the generalizability by aggregating datasets? We find a multi dataset model built on the aggregation of datasets has an improved performance (avg F1=0.69). (3) Given a model built on multiple datasets, how much additional data is required to fine-tune it? We find it challenging to ascertain a minimum number of data points due to the lack of pattern in performance. Investigating possible reasons for the choppy model performance we find that texts are not easily differentiable by stances, nor are annotations consistent within and across datasets. Our observations emphasize the need for an aggregated dataset as well as consistent labels for the generalizability of models.

A multi-task attention tree neural net for stance classification and rumor veracity detection

A multi-task attention tree neural net for stance classification and rumor veracity detection

Similar Word Replacement Method for Improving News Commenter Analysis

In Korea, it is common to read and comment on news stories on portal sites. To influence public opinion, some people write comments repeatedly, some of which are similar to those posted by others. This has become a serious social issue. In our previous research, we collected approximately 2.68 million news comments posted in April 2017. We classified the political stance of each author using a deep learning model (seq2seq), and evaluated how many similar comments each user wrote, as well as how similar each comment was to those posted by other people, using the Jaccard similarity coefficient. However, as our previous model used Jaccard’s similarity only, the meaning of the comments was not considered. To solve this problem, we propose similar word replacement (SWR) using word2vec and a method to analyze the similarity between user comments and classify the political stance of each user. In this study, we showed that when our model used SWR rather than Jaccard’s similarity, its ability to detect similarity between comments increased 3.2 times, and the accuracy of political stance classification improved by 6%.

An Explainable Fake News Detector Based on Named Entity Recognition and Stance Classification Applied to COVID-19

Over the last few years, the phenomenon of fake news has become an important issue, especially during the worldwide COVID-19 pandemic, and also a serious risk for the public health. Due to the huge amount of information that is produced by the social media such as Facebook and Twitter it is becoming difficult to check the produced contents manually. This study proposes an automatic fake news detection system that supports or disproves the dubious claims while returning a set of documents from verified sources. The system is composed of multiple modules and it makes use of different techniques from machine learning, deep learning and natural language processing. Such techniques are used for the selection of relevant documents, to find among those, the ones that are similar to the tested claim and their stances. The proposed system will be used to check medical news and, in particular, the trustworthiness of posts related to the COVID-19 pandemic, vaccine and cure.

Rumor Stance Classification in Online Social Networks: The State-of-the-Art, Prospects, and Future Challenges

The emergence of the Internet as a ubiquitous technology has facilitated the rapid evolution of social media as the leading virtual platform for communication, content sharing, and information dissemination. In spite of revolutionizing the way news is delivered to people, this technology has also brought along with itself inevitable demerits. One such drawback is the spread of rumors expedited by social media platforms, which may provoke doubt and fear. Therefore, it is essential to debunk rumors before their widespread use. Over the years, many studies have been conducted to develop effective rumor verification systems. One aspect of such studies focuses on rumor stance classification, which involves the task of utilizing user viewpoints regarding a rumorous post to better predict the veracity of a rumor. Relying on user stances in rumor verification has gained significant importance, for it has resulted in significant improvements in the model performance. In this paper, we conduct a comprehensive literature review of rumor stance classification in complex online social networks (OSNs). In particular, we present a thorough description of these approaches and compare their performances. Moreover, we introduce multiple datasets available for this purpose and highlight their limitations. Finally, challenges and future directions are discussed to stimulate further relevant research efforts.

Misinformation Adoption or Rejection in the Era of COVID-19

The COVID-19 pandemic has led to a misinformation avalanche on social media, which produced confusion and insecurity in netizens. Learning how to automatically recognize adoption or rejection of misinformation about COVID-19 enables the understanding of the effects of exposure to misinformation and the threats it presents. By casting the problem of recognizing misinformation adoption or rejection as stance classification, we have designed a neural language processing system operating on micro-blogs which takes advantage of Graph Attention Networks relying on lexical, emotion, and semantic knowledge to discern the stance of each micro-blog with respect to COVID-19 misinformation. This enabled us not only to obtain promising results, but also allowed us to use a taxonomy of COVID-19 misinformation themes and concerns to characterize the misinformation adoption or rejection that can be best recognized automatically.