AMELIA: A family of multi-task end-to-end language models for argumentation

Argument mining (AM) involves extracting argument components and predicting relations between them to create argumentative graphs, which are essential for applications requiring argumentative comprehension.To automatically provide high-quality graphs, previous works require a large amount of human-annotated training samples to train AM models.Instead, we leverage a large language model (LLM) to assign pseudo-labels to training samples for reducing reliance on human-annotated training data.However, the training data weaklylabeled by the LLM are too noisy to develop an AM model with reliable performance.In this paper, to improve the model performance, we propose a center-based component detector that refines the boundaries of the detected components and a relation denoiser to deal with noise present in the pseudo-labels when classifying relations between detected components.Experimentally, our AM model improves the boundary detection obtained from the LLM by up to 16% in terms of IoU 75 and of the relation classification obtained from the LLM by up to 12% in terms of macro-F1 score.Our AM model achieves new state-of-the-art performance in weakly-supervised AM, showing up to a 6% improvement over the state-of-the-art component detector and up to a 7% improvement over the state-of-the-art relation classifier.Additionally, our model uses less than 20% of human-annotated data to match the performance of state-of-the-art fully-supervised AM models.

Mining an argument structure from text is an important step for tasks such as argument search and summarization. While studies on argument(ation) mining have proposed promising neural network models, they usually suffer from a shortage of training data. To address this issue, we expand the training data with various auxiliary argument mining corpora and propose an end-to-end cross-corpus training method called Multi-Task Argument Mining (MT-AM). To evaluate our approach, we conducted experiments for the main argument mining tasks on several well-established argument mining corpora. The results demonstrate that MT-AM generally outperformed the models trained on a single corpus. Also, the smaller the target corpus was, the better the MT-AM performed. Our extensive analyses suggest that the improvement of MT-AM depends on several factors of transferability among auxiliary and target corpora.

This study introduces CR‐M‐SpanBERT, a coreference resolution (CR) model that utilizes multiple embedding‐based span bidirectional encoder representations from transformers, for antecedent recognition in natural language (NL) text. Information extraction studies aimed to extract knowledge from NL text autonomously and cost‐effectively. However, the extracted information may not represent knowledge accurately owing to the presence of ambiguous entities. Therefore, we propose a CR model that identifies mentions referring to the same entity in NL text. In the case of CR, it is necessary to understand both the syntax and semantics of the NL text simultaneously. Therefore, multiple embeddings are generated for CR, which can include syntactic and semantic information for each word. We evaluate the effectiveness of CR‐M‐SpanBERT by comparing it to a model that uses SpanBERT as the language model in CR studies. The results demonstrate that our proposed deep neural network model achieves high‐recognition accuracy for extracting antecedents from NL text. Additionally, it requires fewer epochs to achieve an average F1 accuracy greater than 75% compared with the conventional SpanBERT approach.

Given a controversial issue, argument mining from texts in natural language is extremely challenging: besides linguistic aspects, domain knowledge is often required together with appropriate forms of inferences to identify arguments. Via the analysis of various corpora, this contribution explores the types of knowledge that are required to develop an efficient argument mining system. We show that the Qualia structure of the Generative Lexicon with some extensions and a specific interpretation has some expressive capabilities which are appropriate for this task.

Argument-based decision making has been employed to support a variety of reasoning tasks over medical knowledge. These include evidence-based justifications of the effects of treatments, the detection of conflicts in the knowledge base, and the enabling of uncertain and defeasible reasoning in the health-care sector. However, a common limitation of these approaches is that they rely on structured input information. Recent advances in argument mining have shown increasingly accurate results in detecting argument components and predicting their relations from unstructured, natural language texts. In this study, we discuss evidence and claim detection from Randomized Clinical Trials. To this end, we create a new annotated dataset about four different diseases (glaucoma, diabetes, hepatitis B, and hypertension), containing 976 argument components (697 containing evidence, 279 claims). Empirical results are promising, and show the portability of the proposed approach over different branches of medicine.

Argumentation Mining (AM), a specialized branch of Natural Language Processing (NLP), which extract arguments from text and mapping out their relationships. While machine learning has been extensively explored for AM sub tasks, there's still a gap in structuring these methods to spot common patterns across different applications. This study, based on a review of 64 research papers, breaks down how AM is applied across various domains ranging from user-generated texts, English texts, speech to debates, legal documents, and scientific or medical texts. Among these, text takes the lead as the most researched area. Particularly Support Vector Machines (SVM), Bidirectional Encoder Representations from Transformers (BERT) and Bidirectional Long Short-Term Memory (BiLSTM), Convolutional Neural Network (CNN) are some machine learning models that dominates the field. The effectiveness of these models varies depending upon the type of text, excelling in user-generated text where as others perform better with scientific or medical data. The study highlights the need to further explore less-researched areas especially machine learning applications in legal, medical scientific and English texts and critically examine how Large language model and deep learning stacks up against traditional methods. By mapping these insights, the goal is to help researchers pick the right approach for specific AM tasks, ultimately pushing the field forward.

In cases, the same or similar network architecture is used to deal with related but different tasks, where tasks come from different statistical distributions in the sample input space and share some common features. Multi-Task Learning (MTL) combines multiple related tasks for training at the same time, so as to learn some shared feature representation among multiple tasks. However, it is difficult to improve each task when statistical distributions of these related tasks are greatly different. This is caused by the difficulty of extracting an effective generalization of feature representation from multiple tasks. Moreover, it also slows down the convergence rate of MTL. Therefore, we propose a MTL method based on the Localized Generalization Error Model (L-GEM). The L-GEM improves the generalization capability of the trained model by minimizing the upper bound of generalization error of it with respect to unseen samples similar to training samples. It also helps to narrow the gap between different tasks due to different statistical distributions in MTL. Experimental results show that the L-GEM speeds up the training process while significantly improves the final convergence results.

Arguments facilitate humans to deliver their ideas. The outcome of the discussion heavily relies on the validity of the argument. If an argument is well-composed, it is more effective to grasp the core idea behind the argument. To grade the argument, machines can be utilized by decomposing into semantic label components. In natural language processing, multiple language models are available to perform this task. It is divided into context-free and contextual models. The majority of previous studies used hand-crafted features to perform argument component classification, while state of the art language models utilize machine learning. The majority of these language models ignore the context in an argument. This research paper aims to analyze whether by including the context in the classification process may improve the accuracy of the language model which will enhance the argumentation mining process as well. The same document corpus is fed into several language models. Word2Vec and GLoVe represent the context free models, while BERT and ELMo as context sensitive language models. Accuracy and time from each model are then compared to determine the importance of context. The result shows that contextual language models are proven to be able to boost classification accuracy by approximately 20%. However, time comes as a cost where contextual models require longer training and prediction time. The benefit from the increase in accuracy outweighs the burden of time. Thus, as a contextual task, argumentation mining is suggested to use contextual model where context must be included to achieve promising results.

This work examines the methods of optimizing Large Language Models, focusing on their applications in present-day implementations of Artificial Intelligence across sectors such as Natural Language Processing, customer support, and text creation. As a result, prior models have restrictions when it comes to processing the more complex linguistic features, proper usage of CPU and those models are not adhering the ethical practices. This research responds to these challenges by exploring advancements in model designomal structure and update methods, together with categorical responsibility frameworks. Measuring the efficiency of large language models and the potential for their optimization, the work also uses the analysis of user feedback. The research shows that small changes in model design and training strategies have high impact on model performance and interpretability in various language tasks. These findings help to advance the creation of improved and more reliable language models with optimised ethical frameworks and utility of resources. Areas for future work are presented, such as integration of more complex multi-modal data and decreasing the amount of bias in model outputs.

Arguments are structures of premises and conclusions that underpin rational reasoning processes. Within complex knowledge domains, especially if they are contentious, argument structures can become large and complex. Visualization tools have been developed that support argument analysts and help them to work with arguments. Until recently, arguments were manually analyzed from natural language text, or constructed from scratch, but new communication modes mean that increasing amounts of debate and the arguments therein can be captured digitally. Furthermore, new tools and techniques for argument mining are beginning to automate the process of extracting argument structure from natural language; leading to much larger argument datasets that present problems for the current generation of argument visualization tools. Additionally, individual argument analysts have different foci which can lead to increased complexity within datasets, and additional facets that argument visualizations should account for but do not. We propose a tool for interacting with argument corpora that enable users to explore and understand the reasoning structure of large-scale arguments. The tool will support a range of interactivity techniques and will help users to explore and analyse large-scale arguments, to more rapidly comprehend complex new problem domains.

We study the problem of online multitask learning for solving multiple related classification tasks in parallel, aiming at classifying every sequence of data received by each task accurately and efficiently. One practical example of online multitask learning is the micro-blog sentiment detection on a group of users, which classifies micro-blog posts generated by each user into emotional or non-emotional categories. This particular online learning task is challenging for a number of reasons. First of all, to meet the critical requirements of online applications, a highly efficient and scalable classification solution that can make immediate predictions with low learning cost is needed. This requirement leaves conventional batch learning algorithms out of consideration. Second, classical classification methods, be it batch or online, often encounter a dilemma when applied to a group of tasks, i.e., on one hand, a single classification model trained on the entire collection of data from all tasks may fail to capture characteristics of individual task; on the other hand, a model trained independently on individual tasks may suffer from insufficient training data. To overcome these challenges, in this paper, we propose a collaborative online multitask learning method, which learns a global model over the entire data of all tasks. At the same time, individual models for multiple related tasks are jointly inferred by leveraging the global model through a collaborative online learning approach. We illustrate the efficacy of the proposed technique on a synthetic dataset. We also evaluate it on three real-life problems—spam email filtering, bioinformatics data classification, and micro-blog sentiment detection. Experimental results show that our method is effective and scalable at the online classification of multiple related tasks.

We evaluate two large language models (LLMs) ability to perform argumentative reasoning. We experiment with argument mining (AM) and argument pair extraction (APE), and evaluate the LLMs’ ability to recognize arguments under progressively more abstract input and output (I/O) representations (e.g., arbitrary label sets, graphs, etc.). Unlike the well-known evaluation of prompt phrasings, abstraction evaluation retains the prompt’s phrasing but tests reasoning capabilities. We find that scoring-wise the LLMs match or surpass the SOTA in AM and APE, and under certain I/O abstractions LLMs perform well, even beating chain-of-thought–we call this symbolic prompting. However, statistical analysis on the LLMs outputs when subject to small, yet still human-readable, alterations in the I/O representations (e.g., asking for BIO tags as opposed to line numbers) showed that the models are not performing reasoning. This suggests that LLM applications to some tasks, such as data labelling and paper reviewing, must be done with care.

Non-neural approaches to argument mining (AM) are often pipelined and require heavy feature-engineering. In this paper, we propose a neural end-to-end approach to AM which is based on dependency parsing, in contrast to the current state-of-the-art which relies on relation extraction. Our biaffine AM dependency parser significantly outperforms the state-of-the-art, performing at F1 = 73.5% for component identification and F1 = 46.4% for relation identification. One of the advantages of treating AM as biaffine dependency parsing is the simple neural architecture that results. The idea of treating AM as dependency parsing is not new, but has previously been abandoned as it was lagging far behind the state-of-the-art. In a thorough analysis, we investigate the factors that contribute to the success of our model: the biaffine model itself, our representation for the dependency structure of arguments, different encoders in the biaffine model, and syntactic information additionally fed to the model. Our work demonstrates that dependency parsing for AM, an overlooked idea from the past, deserves more attention in the future.

Stance can be defined as positively or negatively evaluating persons, things, or ideas (Du Bois, 2007). Understanding the stance that people express through social media has several applications: It allows governments, companies, or other information seekers to gain insights into how people evaluate their ideas or products. Being aware of the stance of others also enables social media users to engage in discussions more efficiently, which may ultimately lead to better collective decisions. Since the volume of social media posts is too large to be analyzed manually, computeraided methods for understanding stance are necessary. In this thesis, we study three major aspects of such computer-aided methods: (i) abstract formalizations of stance which we can quantify across multiple social media posts, (ii) the creation of suitable datasets that correspond to a certain formalization, and (iii) stance detection systems that can automatically assign stance labels to social media posts. We examine four different formalizations that differ in how specific the insights and supported use-cases are: Stance on Single Targets defines stance as a tuple consisting of a single target (e.g. Atheism) and a polarity (e.g. being in favor of the target), Stance on Multiple Targets models a polarity expressed towards an overall target and several logically linked targets, and Stance on Nuanced Targets is defined as a stance towards all texts in a given dataset. Moreover, we study Hateful Stance, which models whether a post expresses hatefulness towards a single target (e.g. women or refugees). Machine learning-based systems require training data that is annotated with stance labels. Since annotated data is not readily available for every formalization, we create our own datasets. On these datasets, we perform quantitative analyses, which provide insights into how reliable the data is, and into how social media users express stance. Our analysis shows that the reliability of datasets is affected by subjective interpretations and by the frequency with which targets occur. Additionally, we show that the perception of hatefulness correlates with the personal stance of the annotators. We conclude that stance annotations are, to a certain extent, subjective and that future attempts on data creation should account for this subjectivity. We present a novel process for creating datasets that contain subjective stances towards nuanced assertions and which provide comprehensive insights into debates on controversial issues. To investigate the state-of-the-art of stance detection methods, we organized and participated in relevant shared tasks, and conducted experiments on our own datasets. Across all datasets, we find that comparatively simple methods yield a competitive performance. Furthermore, we find that neuronal approaches are competitive, but not clearly superior to more traditional approaches on text classification. We show that approaches based on judgment similarity – the degree to which texts are judged similarly by a large number of people – outperform reference approaches by a large margin. We conclude that judgment similarity is a promising direction to achieve improvements beyond the state-of-the-art in automatic stance detection and related tasks such as sentiment analysis or argument mining.

Available corpora for Argument Mining differ along several axes, and one of the key differences is the presence (or absence) of discourse markers to signal argumentative content. Exploring effective ways to use discourse markers has received wide attention in various discourse parsing tasks, from which it is well-known that discourse markers are strong indicators of discourse relations. To improve the robustness of Argument Mining systems across different genres, we propose to automatically augment a given text with discourse markers such that all relations are explicitly signaled. Our analysis unveils that popular language models taken out-of-the-box fail on this task; however, when fine-tuned on a new heterogeneous dataset that we construct (including synthetic and real examples), they perform considerably better. We demonstrate the impact of our approach on an Argument Mining downstream task, evaluated on different corpora, showing that language models can be trained to automatically fill in discourse markers across different corpora, improving the performance of a downstream model in some, but not all, cases. Our proposed approach can further be employed as an assistant tool for better discourse understanding.