Contradiction Detection Research Articles

Automated requirement contradiction detection through formal logic and LLMs

This paper introduces ALICE (Automated Logic for Identifying Contradictions in Engineering), a novel automated contradiction detection system tailored for formal requirements expressed in controlled natural language. By integrating formal logic with advanced large language models (LLMs), ALICE represents a significant leap forward in identifying and classifying contradictions within requirements documents. Our methodology, grounded on an expanded taxonomy of contradictions, employs a decision tree model addressing seven critical questions to ascertain the presence and type of contradictions. A pivotal achievement of our research is demonstrated through a comparative study, where ALICE’s performance markedly surpasses that of an LLM-only approach by detecting 60% of all contradictions. ALICE achieves a higher accuracy and recall rate, showcasing its efficacy in processing real-world, complex requirement datasets. Furthermore, the successful application of ALICE to real-world datasets validates its practical applicability and scalability. This work not only advances the automated detection of contradictions in formal requirements but also sets a precedent for the application of AI in enhancing reasoning systems within product development. We advocate for ALICE’s scalability and adaptability, presenting it as a cornerstone for future endeavors in model customization and dataset labeling, thereby contributing a substantial foundation to requirements engineering.

Towards an automatic contradiction detection in requirements engineering

AbstractThis paper presents a novel method for automatic contradiction detection in requirements engineering using a hybrid approach combining formal logic with Large Language Models (LLMs), specifically GPT-3. Our three-phase process detects contradictions by identifying conditionals and pseudo-grammatical elements, and employing LLMs for nuanced contradiction detection. Tested extensively, including on a real-world electric bus project, our method achieved 99% accuracy and 60% recall. This approach significantly reduces manual effort, enhances quality, and is scalable for future advancements.

AUTOMATED CONDITION DETECTION IN REQUIREMENTS ENGINEERING

AbstractIn product development, it is of great importance that a complete, unambiguous, and, as far as possible, contradiction-free target system is defined. Requirements documents of complex systems can contain several thousand individual requirements, derived in an interdisciplinary manner and written in natural language by many different stakeholders. Hence, errors, in the form of contradictions, cannot be completely avoided in these documents and today they must be corrected manually with high effort.This paper presents an important building block for automated contradiction detection and quality analysis of requirements documents. We discuss the necessary identification of conditions in requirements and the extraction of the verbal expressions associated with condition and effect, respectively. We applied and analyzed natural language processing methods based on grammatical versus machine learning models. The models have been applied to 1,861 real-world requirements. Both approaches generate promising results, with an accuracy partly over 98%. However, in structured specification texts, a grammatical model is preferable due to lower effort in preprocessing and better usability.

Camouflaged objects detection network via contradiction detection and feature aggregation

Camouflaged Object Detection(COD) aims to segment objects with a similar appearance to the background. There are some problems in existing algorithms, such as blurred edges and incomplete detection. To address the above issues, we propose a novel COD framework termed CFNet. Our network consists of the Contradiction Area Detection Module(CADM) and Feature Aggregation Module(FAM). In the CADM, we propose an improved receptive field mechanism, which utilizes max pooling operation and convolution block to highlight the contradictory areas and refine the edge of the hidden object. Besides, the FAM is designed to connect two adjacent layers via attention and anti-attention strategy, leading to cross-layer feature enhancement and information fusion. More specifically, the self-attention mechanism helps supplement semantic information, and the anti-attention mechanism contributes to removing redundant information. Extensive experiments conducted on the four public COD datasets show the comparable performance of the proposed CFNet with SOTAs, and the ablation experiments demonstrate the effectiveness of the proposed modules.

Detecting contradictions from IoT protocol specification documents based on neural generated knowledge graph

Due to the boom of Internet of Things (IoT) in recent years, various IoT devices are connected to the Internet and communicate with each other through network protocols such as the Constrained Application Protocol (CoAP). These protocols are typically defined and described in specification documents, such as Request for Comments (RFC), which are written in natural or semi-formal languages. Since developers largely follow the specification documents when implementing web protocols, they have become the de facto protocol specifications. Therefore, it must be ensured that the descriptions in them are consistent to avoid technological issues, incompatibility, security risks, or even legal concerns. In this work, we propose Neural RFC Knowledge Graph (NRFCKG), a neural network-generated knowledge graph based contradictions detection tool for IoT protocol specification documents. Our approach can automatically parse the specification documents and construct knowledge graphs from them through entity extraction, relation extraction, and rule extraction with large language models. It then conducts an intra-entity and inter-entity contradiction detection over the generated knowledge graph. We implement NRFCKG and apply it to the most extensively used messaging protocols in IoT, including the main RFC (RFC7252) of CoAP, the specification document of MQTT, and the specification document of AMQP. Our evaluation shows that NRFCKG generalizes well to other specification documents and it manages to detect contradictions from these IoT protocol specification documents.

BM5-SP-SC: A Dual Model Architecture for Contradiction Detection on Crowdfunding Projects

Despite the prevalence of scams in crowdfunding projects, currently, there is little research into the identification of fraudulent or infeasible crowdfunding projects. Since detecting fraudulent crowdfunding projects is challenging, most existing research on fake information has focused on detecting fake news or fake charity crowdfunding projects based on social media, but research on fraudulent or infeasible crowdfunding projects is very limited. Therefore, to solve this problem, we focus on how to detect fraudulent crowdfunding projects based on knowledge extraction and contradiction detection. We proposed a novel method called BM5-SP-SC (BERT-MT5-Sentence Pattern-Sentiment Classification). BM5 (BERT-MT5), which is built from a combination of a key-BERT and a fine-tuned MT5 transformers, was used to extract feature information from crowdfunding projects. We proposed a novel method for MT5 training to construct an adaptive BM5 model. The correct rate of keywords extracted by our novel adaptive BM5 model was up to 72.7%, the recall was 100%, and the F-measure was up to 84.19%. The minimum train loss of the BM5 model was up to 0.1342, and the evaluation loss achieved was 0.3064. The BLEU score of summary-to-keyword was 37.336. Moreover, we proposed an SP (Sentence Pattern) matching method to achieve knowledge extraction. Furthermore, SC (Sentiment Classification) was used to build a sentiment classifier thesaurus for identifying fraudulent and infeasible crowdfunding projects. Our proposed BM5-SP-SC achieved an overall accuracy of 85.26% in detecting fraudulent crowdfunding projects.

A Neuro Symbolic Approach for Contradiction Detection in Persian Text

Detection of semantic contradictory sentences is a challenging and fundamental issue for some NLP applications, such as textual entailments recognition. In this study, contradiction means different types of semantic confrontation, such as negation, antonymy, and numerical. Due to the lack of sufficient data to apply precise machine learning and, specifically, deep learning methods to Persian and other low-resource languages, rule-based approaches are of great interest. Also, recently, the emergence of new methods such as transfer learning has opened up the possibility of deep learning for low-resource languages. This paper introduces a hybrid contradiction detection approach for detecting seven categories of contradictions in Persian texts: Antonymy, negation, numerical, factive, structural, lexical and world knowledge. The proposed method consists of 1) a novel data mining method and 2) a transformer-based deep neural method for contradiction detection . Also, a simple baseline is presented for comparison. The data mining method uses frequent rule mining to extract appropriate contradiction detection rules employing a development set. Extracted rules are tested for different categories of contradictory sentences. In the first step, a classifier checks whether the rules work for an input sentence pair. Then, according to the result, rules are used for three categories of negation, numerical, and antonym. In this part, the highest F-measure is obtained for detecting the negation category (90%), the average F-measure for these three categories is 86%, and for the other four categories, in which the rules have a lower F-measure of 62%, the transformer-based method achieved 76%. The proposed hybrid approach has an overall f-measure of higher than 80%. 

Editorial

Dear Readers,  It gives me great pleasure to announce the third regular issue of 2023. In this issue, 4 papers by 11 authors from 5 countries cover various topical aspects of computer science. In an ongoing effort to further strengthen our journal, I would like to expand the editorial board: If you are a tenured associate professor or above with a strong publication record, you are welcome to apply to join our editorial board. We are also interested in high-quality proposals for special issues on new topics and trends.  As always, I would like to thank all the authors for their sound research and the editorial board for their extremely valuable review effort and suggestions for improvement. These contributions, together with the generous support of the consortium members, sustain the quality of our journal.  In the third regular issue, I am very pleased to introduce the following 4 accepted articles: Gerardo Matturro from Uruguay reports on his research findings on undergraduate software engineering over a seven-year period, specifically on students' motivation to participate in research projects, skills acquired, and their perceptions of benefits. Uma Priya D and P. Santhi Thilagam from India propose an approach to cluster heterogeneous JSON documents using the similarity fusion method based on structural, semantic and contextual measures of JSON schemas. Zeinab Rahimi and Mehrnoush Shamsfard from Iran present a hybrid contradiction detection approach that can detect seven categories of contradictions in Persian texts: Antonymy, negation, numerical, factive, structural, lexical and world knowledge, which is based on a novel data mining method and a transformer-based deep neural method for contradiction detection. In their joint research between Estonia, Spain and India, Shashi Kant Shankar, Adolfo Ruiz-Calleja, Luis P. Prieto, María Jesús Rodríguez-Triana, Pankaj Chejara, and Sandesh Tripathi discuss a modular and modifiable infrastructure for data preparation, organization, and fusion to partially support the development of context-aware multimodal learning analytics solutions.  Enjoy Reading! Cordially, 

ARNLI: ARABIC NATURAL LANGUAGE INFERENCE ENTAILMENT AND CONTRADICTION DETECTION

Natural Language Inference (NLI) is a hot topic research in natural language processing, contradiction detection between sentences is a special case of NLI. This is considered a difficult NLP task which has a big influence when added as a component in many NLP applications, such as Question Answering Systems, text Summarization. Arabic Language is one of the most challenging low-resources languages in detecting contradictions due to its rich lexical, semantics ambiguity. We have created a dataset of more than 12k sentences and named ArNLI, that will be publicly available. Moreover, we have applied a new model inspired by Stanford contradiction detection proposed solutions on English language. We proposed an approach to detect contradictions between pairs of sentences in Arabic language using contradiction vector combined with language model vector as an input to machine learning model. We analyzed results of different traditional machine learning classifiers and compared their results on our created dataset (ArNLI) and on an automatic translation of both PHEME, SICK English datasets. Best results achieved using Random Forest classifier with an accuracy of 99%, 60%, 75% on PHEME, SICK and ArNLI respectively.

Contradiction Detection in Financial Reports

Finding and amending contradictions in a financial report is crucial for the publishing company and its financial auditors. To automate this process, we introduce a novel approach that incorporates informed pre-training into its transformer-based architecture to infuse this model with additional Part-Of-Speech knowledge. Furthermore, we fine-tune the model on the public Stanford Natural Language Inference Corpus and our proprietary financial contradiction dataset. It achieves an exceptional contradiction detection F1 score of 89.55% on our real-world financial contradiction dataset, beating our several baselines by a considerable margin. During the model selection process we also test various financial-document-specific transformer models and find that they underperform the more general embedding approaches.

Fundamental Research on Detecting Contradictions in Requirements: Taxonomy and Semi-Automated Approach

Requirements documents can contain several thousand individual requirements. They must be error-free to avoid unnecessary complications and costs in the later product development stages. An important part of this is to identify contradictions between two requirements. The first step is therefore to define what contradictions are and in what form they can occur in requirement documents. In this paper the scientific theories regarding contradictions are discussed, concerning to their usefulness for the topic. In doing so, the Aristotelian Logic proved to provide the best basis for an application in the Requirements Engineering context. Based on this theory, we have created specific subtypes of contradictions to match them to the requirements engineering field. The identification of these subtypes is done by a formalization of the requirement sentences and a subsequent analysis by means of simple questions. To validate the method, industrial requirement documents were searched for contradictions. For each detected type of contradiction, we present an example of the detection process. Thereby, we show that the method is easy to apply and may also be used by non-specialists. Thus, our method provides a taxonomy as a basis for further research on automated contradiction detection as well as on automated quality analysis of requirements documents.

Contradiction Detection and Repair in a Large Theory

As with any software, the challenges of developing large andmanually-created axiomatizations in an expressive logic suchas first order logic with equality can be very different fromthose found in comparatively small theories. We present someof the tools and practices that have supported development ofa logical theories with tens of thousands of statements, andensured that they are free of logical contradiction, and suit-able for automated theorem reasoning.

Product online review analysis using fuzzy ontology

AbstractThis paper analyses the online text comments in the online trading platform and identifies the abnormal and contradictory comments. Firstly, the abnormal contradiction detection and analysis model of online text reviews is constructed based on fuzzy ontology to identify the abnormal and contradictory online text reviews. Secondly, the fuzzy emotional ontology and emotional tendency analysis method identify abnormal and contradictory online text reviews. The online text review anomaly detection and analysis model constructed enriches the process and method of online review anomaly identification.

AN EXPERIMENTAL EVALUATION OF DEEP NEURAL NETWORK MODEL PERFORMANCE FOR THE RECOGNITION OF CONTRADICTORY MEDICAL RESEARCH CLAIMS USING SMALL AND MEDIUM-SIZED CORPORA

Corpora come in various shapes and sizes and play an essential role in facilitating Natural Language Processing (NLP) tasks. However, the availability of corpora specialized for Evidence-Based Medicine (EBM) related tasks is limited. The study is aimed to discover how the size of a corpus influence the performance of our Deep Neural Network (DNN) model developed for contradiction detection in medical literature. We explored the potential of the EBM Summarizer corpus by Mollá and Santiago-Martínez, a medium-sized corpus to be used with our contradiction detection model. The dataset preparation involves the filtering of open-ended questions, duplicates of claims, and vague claims. As a result, two datasets were created with the claim input represented by sniptext in one dataset and longtext in the other. Experiments were conducted with varying numbers of hidden layers and units of the model using different datasets. The performance of the DNN model was recorded and compared with the result of using a small-sized corpus. It was found that the DNN model performance did not improve even after it was trained with a larger dataset derived from the medium-sized corpus. The factors may include the limitation of the DNN model itself and the quality of the datasets.

Sentiment Analysis of Students' Feedback in MOOCs: A Systematic Literature Review.

In recent years, sentiment analysis (SA) has gained popularity among researchers in various domains, including the education domain. Particularly, sentiment analysis can be applied to review the course comments in massive open online courses (MOOCs), which could enable instructors to easily evaluate their courses. This article is a systematic literature review on the use of sentiment analysis for evaluating students’ feedback in MOOCs, exploring works published between January 1, 2015, and March 4, 2021. To the best of our knowledge, this systematic review is the first of its kind. We have applied a stepwise PRISMA framework to guide our search process, by searching for studies in six electronic research databases (ACM, IEEE, ScienceDirect, Springer, Scopus, and Web of Science). Our review identified 40 relevant articles out of 440 that were initially found at the first stage. From the reviewed literature, we found that the research has revolved around six areas: MOOC content evaluation, feedback contradiction detection, SA effectiveness, SA through social network posts, understanding course performance and dropouts, and MOOC design model evaluation. In the end, some recommendations are provided and areas for future research directions are identified.

“Here Are the Rules: Ignore All Rules”: Automatic Contradiction Detection in Spanish

This paper tackles automatic detection of contradictions in Spanish within the news domain. Two pieces of information are classified as compatible, contradictory, or unrelated information. To deal with the task, the ES-Contradiction dataset was created. This dataset contains a balanced number of each of the three types of information. The novelty of the research is the fine-grained annotation of the different types of contradictions in the dataset. Presently, four different types of contradictions are covered in the contradiction examples: negation, antonyms, numerical, and structural. However, future work will extend the dataset with all possible types of contradictions. In order to validate the effectiveness of the dataset, a pretrained model is used (BETO), and after performing different experiments, the system is able to detect contradiction with a F1m of 92.47%. Regarding the type of contradictions, the best results are obtained with negation contradiction (F1m = 98%), whereas structural contradictions obtain the lowest results (F1m = 69%) because of the smaller number of structural examples, due to the complexity of generating them. When dealing with a more generalistic dataset such as XNLI, our dataset fails to detect most of the contradictions properly, as the size of both datasets are very different and our dataset only covers four types of contradiction. However, using the classification of the contradictions leads us to conclude that there are highly complex contradictions that will need external knowledge in order to be properly detected and this will avoid the need for them to be previously exposed to the system.

Towards an Enhanced Aspect-based Contradiction Detection Approach for Online Review Content

User generated content as such online reviews plays an important role in customer’s purchase decisions. Many works have focused on identifying satisfaction of the reviewer in social media through the study of sentiment analysis (SA) and opinion mining. The large amount of potential application and the increasing number of opinions expresses on the web results in researchers interest on sentiment analysis and opinion mining. However, due to the reviewer’s idiosyncrasy, reviewer may have different preferences and point of view for a particular subject which in this case hotel reviews. There is still limited research that focuses on this contradiction detection in the perspective of tourism online review especially in numerical contradiction. Therefore, the aim of this paper to investigate the type of contradiction in online review which mainly focusing on hotel online review, to provide useful material on process or methods for identifying contradiction which mainly on the review itself and to determine opportunities for relevant future research for online review contradiction detection. We also proposed a model to detect numerical contradiction in user generated content for tourism industry.

Contradiction Detection with Contradiction-Specific Word Embedding

Contradiction detection is a task to recognize contradiction relations between a pair of sentences. Despite the effectiveness of traditional context-based word embedding learning algorithms in many natural language processing tasks, such algorithms are not powerful enough for contradiction detection. Contrasting words such as “overfull” and “empty” are mostly mapped into close vectors in such embedding space. To solve this problem, we develop a tailored neural network to learn contradiction-specific word embedding (CWE). The method can separate antonyms in the opposite ends of a spectrum. CWE is learned from a training corpus which is automatically generated from the paraphrase database, and is naturally applied as features to carry out contradiction detection in SemEval 2014 benchmark dataset. Experimental results show that CWE outperforms traditional context-based word embedding in contradiction detection. The proposed model for contradiction detection performs comparably with the top-performing system in accuracy of three-category classification and enhances the accuracy from 75.97% to 82.08% in the contradiction category.

Detection of contradictions by relation matching and uncertainty assessment

Contradiction detection is a difficult task in the field of natural language processing given the variety of ways contradictions occur across texts. If blunt negations, antonyms and numerical mismatches are obvious features to convey contradictions, they also arise from inconsistent domain knowledge, uncertain co-references or differences in the structures of assertions. In this paper, we investigate the problem of contradictions detection for uncertain statements when the author provides not only factual information but also clues about its plausibility. The problem is of particular interest for application fields relying on reported information, when decision makers receive information from various sources. Along with hints as to the derivation of the content, authors often embed clues as to how strong they support reported facts, in the form on confidence, skepticism, doubt or strong conviction. For such assertions, contradictions highlight not only impossible versions of reported events and actions but also discrepancies in the assessment of their plausibility. After analyzing various types of contradictions in subjective statements, we describe a model to detect contradictions thanks to a joint analysis of functional relations and uncertainty assessments.

Computational mental simulation: A review

AbstractThis paper is dedicated to the study of existing approaches that explicitly use mental simulation. Current implementations of the mental simulation paradigm, taken together, computationally address many aspects suggested by cognitive science research. Agents are able to find solutions to nontrivial scenarios in virtual or physical environments. Existing systems also learn new behavior by imitation of others similar to them and model the behavior of different others with the help of specialized models, culminating with the collaboration between agents and humans. Approaches that use self models are able to mentally simulate interaction and to learn about their own physical properties. Multiple mental simulations are used to find solutions to tasks, for truth maintenance, and contradiction detection. However, individual approaches do not cover all of the contexts of mental simulation and most rely on techniques which are only suitable for subsets of obtainable functionality. This review spans through four perspectives on the functionality of state‐of‐the‐art artificial intelligence applications, while linking them to cognitive science research results. Finally, an overview identifies the main gaps in existing literature on computational mental simulation and provides our suggestions for future development.