Recognizing Textual Entailment Research Articles

Towards Improvement of Grounded Cross-lingual Natural Language Inference with VisioTextual Attention

Natural Language Inference (NLI) has been one of the fundamental tasks in Natural Language Processing (NLP). Recognizing Textual Entailment (RTE) between the two pieces of text is a crucial problem. It adds further challenges when it involves two languages, i.e., in the cross-lingual scenario. In this paper, we propose VisioTextual-Attention (VTA) — an effective visual–textual coattention mechanism for multi-modal cross-lingual NLI. Through our research, we show that instead of using only linguistic input features, introducing visual features supporting the textual inputs improves the performance of NLI models if an effective cross-modal attention mechanism is carefully constructed. We perform several experiments on a standard cross-lingual textual entailment dataset in Hindi–English language pairs and show that the addition of visual information to the dataset along with our proposed VisioTextual Attention (VTA) enhances performance and surpasses the current state-of-the-art by 4.5%. Through monolingual experiments, we also show that the proposed VTA mechanism surpasses monolingual state-of-the-art by a margin of 2.89%. We argue that our VTA mechanism is model agnostic and can be used with other deep learning-based architectures for grounded cross-lingual NLI.

VLSP 2021 - vnNLI Challenge: Vietnamese and English-Vietnamese Textual Entailment

This paper presents the first challenge on recognizing textual entailment (RTE), also known as natural language inference (NLI), held in a Vietnamese Language and Speech Processing workshop (VLSP 2021).The challenge aims to determine, for a given pair of sentences, whether the two sentences semantically agree, disagree, or are neutral/irrelevant to each other. The input sentences are in English or Vietnamese and may not be in the same language. This task is important in identifying, from different information sources, the evidence that supports or refutes a statement. The identification of such evidence is subsequently useful for many information tracking applications, such as opinion mining, brand and reputation management, and particularly fighting against fake news.Through this challenge, we would like to provide an opportunity for participants who are interested in the problem, to contribute their knowledge to improve the existing techniques and methods for the task, so as to enhance the effectiveness of those applications.In the paper, we introduce a collection of Vietnamese and English sentences in the domain of health that we built to serve as a benchmarking dataset for the task. We also describe the evaluation results of systems participating in the challenge.

Feature-Rich Classifiers for Recognizing Textual Entailment in Indonesian

Recognizing Textual Entailment (RTE) is a Natural Language Processing task to determine whether a sentence (text) semantically entails another sentence (hypothesis). In this paper, we extracted and learned 35 features from a pair of text and hypothesis in Indonesian. The ablation study was conducted to analyze features contribution to RTE model. The experiments shown that, using Support Vector Machine (SVM) and Logistic Regression, the token-based features contribute positively to improve the model performance. The best model in our experiment is SVM that scored F1-Score of 79.65%. Despite sacrificing 5-points accuracy to the state-of-the-art BERT model, SVM classifier is 31 hours more efficient in terms of training time.

Using Earth Mover's Distance and Machine Learning for Recognizing Textual Entailment in Arabic

Recognizing Textual Entailment (RTE) is a task of Natural Language Processing (NLP) in which two texts denoted TEXT (T) and HYPOTHESIS (H) are processed by a system to determine whether the meaning of H is inferred (entailed) from T or not. This task is useful for several NLP applications and it has attracted a lot of attention in research. Most of studies focused on English as a target language. In this paper, we give an overview of the main studies on Textual Entailment for English and Arabic and we present a new approach to deal with this task for Arabic using a measure of similarity based on Earth Mover's Distance, and machine learning. We experimented this approach using state of the art Arabic NLP tools and we achieved encouraging results. Although we have applied this approach only to Arabic, its application to other languages is still possible.

DGI: Recognition of Textual Entailment via dynamic gate Matching

Recognizing Textual Entailment (RTE) is an integral part of intelligent machines which is able to understand and reason with natural languages. Some special embedding methods such as the attention mechanism exploit the semantic information without considering the features of sentence interaction, which also affect the word-level attention weight when a word appears at multiple positions of a sentence. In this study, we propose a Dynamic Gate Inference model (DGI) to fulfill the RTE task. In the DGI model, different aspects of semantic information are extracted from a premise sentence and a hypothesis sentence by a proposed dynamic gate Matching LSTM structure (gMatch), which combines the word-level fine-grained reasoning mechanism with the sentence-level gating structure to capture the global semantics. The textual relationship between the premise and the hypothesis is inferred by the three categories of attention including direct concatenation, similarity and difference. Extensive experiments were conducted to evaluate the performance of the proposed DGI model in two popular corpus by the metric of accuracy, and the results demonstrate that our approach outperforms the state-of-the-art baseline models in textual entailment in an effective manner.

Visual-Textual Hybrid Sequence Matching for Joint Reasoning.

Reasoning is one of the central topics in artificial intelligence. As an important reasoning paradigm, entailment recognition has attracted much research interest, which judges if a hypothesis can be inferred from given premises. However, existing research mainly focuses on text-based analysis, that is, recognizing textual entailment (RTE), which limits its depth and width. Actually, the knowledge and inference process of human are across different sensory organs like language and vision, with unique perspectives to represent complementary reasoning cues. It is significant to extend existing entailment recognition research to cross-media scenarios, that is, recognizing cross-media entailment (RCE). Therefore, this article focuses on one representative RCE task: visual-textual reasoning, and proposes the visual-textual hybrid sequence matching (VHSM) approach. VHSM can reason from image-text premises to text hypotheses, whose contributions are: 1) visual-textual hybrid multicontext inference is proposed to address RCE via matching with hybrid context embeddings, along with adaptive gated aggregation to obtain the final prediction results. It can fully exploit complementary visual-textual cue interaction during joint reasoning; 2) memory attention-based context embedding is proposed to sequentially encode hybrid context embeddings, with the memory attention networks to compare neighboring time-steps. This can capture the important memory dimensions by coefficient assignment, which fully exploits the visual-textual context correlation; and 3) cross-task and visual-textual transfer strategy is further proposed to enrich correlation training information for boosting reasoning accuracy, which transfers knowledge not only from cross-media retrieval task to RCE but also between corresponding text and image premises. The experimental results of recognizing visual-textual entailment task on the SNLI dataset verify the effectiveness of VHSM.

Inherent Disagreements in Human Textual Inferences

We analyze human’s disagreements about the validity of natural language inferences. We show that, very often, disagreements are not dismissible as annotation “noise”, but rather persist as we collect more ratings and as we vary the amount of context provided to raters. We further show that the type of uncertainty captured by current state-of-the-art models for natural language inference is not reflective of the type of uncertainty present in human disagreements. We discuss implications of our results in relation to the recognizing textual entailment (RTE)/natural language inference (NLI) task. We argue for a refined evaluation objective that requires models to explicitly capture the full distribution of plausible human judgments.

DTC: Transfer learning for commonsense machine comprehension

Commonsense Machine Comprehension (CMC) is a popular natural language understanding task. CMC enables computers to learn about causal and temporal reasoning by exploiting implicit commonsense knowledge and can be applied to Question Answering, Search Engine and Dialogue System. Previous methods for CMC limit the vision on CMC task, neglecting that Recognizing Textual Entailment(RTE) task has much similarities with CMC task. In this paper, we propose a transfer learning model, which can take advantage of commonsense knowledge in RTE task by mapping CMC examples and RTE examples to a shared feature space and comprehending in this feature space. Specifically, we first establish a transfer learning framework which has three components: (1) source and target mappings, (2) domain regularization, and (3) CMC score function. Then we make selection for each component in our transfer learning framework and propose the Domain Transfer Comprehension(DTC) model. Experiments on Story Cloze Test show that our model outperforms most previous approaches and provides competitive results with state-of-art methods. We also show each components of our model have positive effect on performance.

Combining Axiom Injection and Knowledge Base Completion for Efficient Natural Language Inference

In logic-based approaches to reasoning tasks such as Recognizing Textual Entailment (RTE), it is important for a system to have a large amount of knowledge data. However, there is a tradeoff between adding more knowledge data for improved RTE performance and maintaining an efficient RTE system, as such a big database is problematic in terms of the memory usage and computational complexity. In this work, we show the processing time of a state-of-the-art logic-based RTE system can be significantly reduced by replacing its search-based axiom injection (abduction) mechanism by that based on Knowledge Base Completion (KBC). We integrate this mechanism in a Coq plugin that provides a proof automation tactic for natural language inference. Additionally, we show empirically that adding new knowledge data contributes to better RTE performance while not harming the processing speed in this framework.

Alignment Based Approach for Arabic Textual Entailment

Recognizing Textual Entailment (RTE) is a task of Natural language processing (NLP) in which two text fragments denoted Text (T) and Hypothesis (H) are processed by a system to determine whether the meaning of H is entailed from T or not. This task is necessary for numerous natural language processing applications. Thus, several approaches have been proposed to deal with the RTE task, running from shallow approaches based on measuring lexical similarity to advanced approaches based on semantic interpretation and deep application of machine learning algorithms. In this paper, we present an approach to deal with Textual Entailment for Arabic language based on machine learning and text alignment modeled as an optimization problem. The aim of this work is to experiment how well state of the art Arabic NLP tools, resources and alignment technique work when applied to Arabic RTE. We report the performance of our system on an existing Arabic RTE dataset and we achieve encouraging results.

Automatic Theorem Proving for Natural Logic: a Case Study on Textual Entailment

Recognizing Textual Entailment (RTE) is a Natural Language Processing task. It is very important in tasks as Semantic Search and Text Summarization. There are many approaches to RTE, for example, methods based on machine learning, linear programming, probabilistic calculus, optimization, and logic. Unfortunately, no one of them can explain why the entailment is carried on. We can make reasonings, with Natural Logic, from the syntactic part of a natural language expression, and very little semantic information. This paper presents an Automatic Theorem Prover for Natural Logic that allows to know precisely the relationships needed in order to reach the entailment in a class of natural language expressions.

Recognizing Textual Entailment: Challenges in the Portuguese Language

Recognizing textual entailment comprises the task of determining semantic entailment relations between text fragments. A text fragment entails another text fragment if, from the meaning of the former, one can infer the meaning of the latter. If such relation is bidirectional, then we are in the presence of a paraphrase. Automatically recognizing textual entailment relations captures major semantic inference needs in several natural language processing (NLP) applications. As in many NLP tasks, textual entailment corpora for English abound, while the same is not true for more resource-scarce languages such as Portuguese. Exploiting what seems to be the only Portuguese corpus for textual entailment and paraphrases (the ASSIN corpus), in this paper, we address the task of automatically recognizing textual entailment (RTE) and paraphrases from text written in the Portuguese language, by employing supervised machine learning techniques. We employ lexical, syntactic and semantic features, and analyze the impact of using semantic-based approaches in the performance of the system. We then try to take advantage of the bi-dialect nature of ASSIN to compensate its limited size. With the same aim, we explore modeling the task of recognizing textual entailment and paraphrases as a binary classification problem by considering the bidirectional nature of paraphrases as entailment relationships. Addressing the task as a multi-class classification problem, we achieve results in line with the winner of the ASSIN Challenge. In addition, we conclude that semantic-based approaches are promising in this task, and that combining data from European and Brazilian Portuguese is less straightforward than it may initially seem. The binary classification modeling of the problem does not seem to bring advantages to the original multi-class model, despite the outstanding results obtained by the binary classifier for recognizing textual entailments.

Building an Evaluation Scale using Item Response Theory.

Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption is that all items in a given test set are equal with regards to difficulty and discriminating power. We propose Item Response Theory (IRT) from psychometrics as an alternative means for gold-standard test-set generation and NLP system evaluation. IRT is able to describe characteristics of individual items - their difficulty and discriminating power - and can account for these characteristics in its estimation of human intelligence or ability for an NLP task. In this paper, we demonstrate IRT by generating a gold-standard test set for Recognizing Textual Entailment. By collecting a large number of human responses and fitting our IRT model, we show that our IRT model compares NLP systems with the performance in a human population and is able to provide more insight into system performance than standard evaluation metrics. We show that a high accuracy score does not always imply a high IRT score, which depends on the item characteristics and the response pattern.

Integrating a semantic-based retrieval agent into case-based reasoning systems: A case study of an online bookstore

Natural language search engines should be developed to provide a friendly environment for business-to-consumer e-commerce that reduce the fatigue customers experience and help them decide what to buy. To support product information retrieval and reuse, this paper presents a novel framework for a case-based reasoning system that includes a collaborative filtering mechanism and a semantic-based case retrieval agent. Furthermore, the case retrieval agent integrates short-text semantic similarity (STSS) and recognizing textual entailment (RTE). The proposed approach was evaluated using competitive methods in the performance of STSS and RTE, and according to the results, the proposed approach outperforms most previously described approaches. Finally, the effectiveness of the proposed approach was investigated using a case study of an online bookstore, and according to the results of case study, the proposed approach outperforms a compared system using string similarity and an existing e-commerce system, Amazon.

Knowledge-Based Textual Inference via Parse-Tree Transformations

Textual inference is an important component in many applications for understanding natural language. Classical approaches to textual inference rely on logical representations for meaning, which may be regarded as "external" to the natural language itself. However, practical applications usually adopt shallower lexical or lexical-syntactic representations, which correspond closely to language structure. In many cases, such approaches lack a principled meaning representation and inference framework. We describe an inference formalism that operates directly on language-based structures, particularly syntactic parse trees. New trees are generated by applying inference rules, which provide a unified representation for varying types of inferences. We use manual and automatic methods to generate these rules, which cover generic linguistic structures as well as specific lexical-based inferences. We also present a novel packed data-structure and a corresponding inference algorithm that allows efficient implementation of this formalism. We proved the correctness of the new algorithm and established its efficiency analytically and empirically. The utility of our approach was illustrated on two tasks: unsupervised relation extraction from a large corpus, and the Recognizing Textual Entailment (RTE) benchmarks.

Recognizing Textual Entailment: Models and Applications Ido Dagan1, Dan Roth2, Mark Sammons2, and Fabio Massimo Zanzotto3 (1Bar-Ilan University, Israel, 2University of Illinois, Urbana, IL, and 3University of Rome “Tor Vergata,” Italy) Morgan & Claypool (Synthesis Lectures on Human Language Technologies, edited by Graeme Hirst, volume 23), 2013, xx+200 pp; paperbound, ISBN 978-1-59829-834-5; e-book, ISBN 978-1-59829-835-2

<b>Recognizing Textual Entailment: Models and Applications</b> <b>Ido Dagan</b>¹<b>, Dan Roth</b>²<b>, Mark Sammons</b>²<b>, and Fabio Massimo Zanzotto</b>³ (¹Bar-Ilan University, Israel, ²University of Illinois, Urbana, IL, and ³University of Rome “Tor Vergata,” Italy) Morgan &amp; Claypool (Synthesis Lectures on Human Language Technologies, edited by Graeme Hirst, volume 23), 2013, xx+200 pp; paperbound, ISBN 978-1-59829-834-5; e-book, ISBN 978-1-59829-835-2

Surface-Similarity Based Textual Entailment Recognition for Japanese Text

This paper proposes a surface-similarity based method for recognizing textual entailment (RTE) in Japanese. First, we experimentally show that there is a positive correlation between semantic similarity (textual entailment) and surface similarity between sentences. The most effective measure of surface similarity for RTE is the character overlap ratio, which achieves classification accuracy of 78.3%. Based on the result, we design a two-step RTE system for binary classification. The first step classifies a given text pair into positive or negative entailment based on the character overlap ratio. If the pair is classified into the positive class, the second step examines whether the assigned class should be flipped or not by using heuristic rules that detect the mismatch of named entities and numbers. In addition to the RTE system, we also implement the MC system that classifies a given text pair into one of four classes (forward entailment, bidirectional entailment, contradiction, and the others), by combining a contradiction detector and the RTE system. In the RITE-2 formal run, the RTE system was ranked 7th among 42 systems at the RTE task, and the MC system was ranked first among 21 systems at the MC task. These results show that the surface-similarity based method achieves high performance in RTE.

Is there a place for logic in recognizing textual entailment?

From a purely theoretical point of view, it makes sense to approach recognizing textual entailment (RTE) with the help of logic. After all, entailment matters are all about logic. In practice, only few RTE systems follow the bumpy road from words to logic. This is probably because it requires a combination of robust, deep semantic analysis and logical inference—and why develop something with this complexity if you perhaps can get away with something simpler? In this article, with the help of an RTE system based on Combinatory Categorial Grammar, Discourse Representation Theory, and first-order theorem proving, we make an empirical assessment of the logic-based approach. High precision paired with low recall is a key characteristic of this system. The bottleneck in achieving high recall is the lack of a systematic way to produce relevant background knowledge. There is a place for logic in RTE, but it is (still) overshadowed by the knowledge acquisition problem.

The BIUTEE Research Platform for Transformation-based Textual Entailment Recognition

Recent progress in research of the Recognizing Textual Entailment (RTE) task shows a constantly-increasing level of complexity in this research field. A way to avoid having this complexity becoming a barrier for researchers, especially for new-comers in the field, is to provide a freely available RTE system with a high level of flexibility and extensibility. In this paper, we introduce our RTE system, BiuTee2, and suggest it as an effective research framework for RTE. In particular, BiuTee follows the prominent transformation-based paradigm for RTE, and offers an accessible platform for research within this approach. We describe each of BiuTee’s components and point out the mechanisms and properties which directly support adaptations and integration of new components. In addition, we describe BiuTee’s visual tracing tool, which provides notable assistance for researchers in refining and “debugging” their knowledge resources and inference components.

Towards a Semantic Model for Textual Entailment Annotation

We introduce a new formal semantic model for annotating textual entailments that describes restrictive, intersective, and appositive modification. The model contains a formally defined interpreted lexicon, which specifies the inventory of symbols and the supported semantic operators, and an informally defined annotation scheme that instructs annotators in which way to bind words and constructions from a given pair of premise and hypothesis to the interpreted lexicon. We explore the applicability of the proposed model to the Recognizing Textual Entailment (RTE) 1–4 corpora and describe a first-stage annotation scheme on which we based the manual annotation work. The constructions we annotated were found to occur in 80.65% of the entailments in RTE 1–4 and were annotated with cross-annotator agreement of 68% on average. The annotated parts of the RTE corpora are publicly available for further research.