Natural Language Inference Datasets Research Articles

Simple Data Transformations for Mitigating the Syntactic Similarity to Improve Sentence Embeddings at Supervised Contrastive Learning

Contrastive learning of sentence representations has achieved great improvements in several natural language processing tasks. However, the supervised contrastive learning model trained on the natural language inference (NLI) dataset is insufficient to elucidate the semantics of sentences since it is prone to make a prediction based on heuristics. Herein, by using the ParsEVAL and the word overlap metric, it is shown that sentence pairs in the NLI dataset have strong syntactic similarity and propose a framework to compensate for this problem in two aspects. 1) Apply simple syntactic transformations to the hypothesis and 2) expand the objective to SupCon Loss to leverage variants of sentences. The method is evaluated on semantic textual similarity (STS) tasks and transfer tasks. The proposed methods improve the performance of the BERT‐based baseline in STS Benchmark and SICK Relatedness by 1.48% and 2.2%. Furthermore, the model achieves 82.65% on the HANS benchmark dataset, to the best of our knowledge, which is a state‐of‐the‐art performance demonstrating that our approach is effective in grasping semantics without heuristics in the NLI dataset at supervised contrastive learning. The code is available at https://github.com/whnhch/Break-the-Similarity.

ArEntail: manually-curated Arabic natural language inference dataset from news headlines

ArEntail: manually-curated Arabic natural language inference dataset from news headlines

Medical education and artificial intelligence: Question answering for medical questions based on intelligent interaction

SummaryComputer assisted medical diagnosis technology is widely used in the field of medical assistance to assist doctors in making diagnostic decisions. But as the number of patients increases, the diagnostic pressure on doctors gradually increases, and more efficient computer‐aided medical diagnosis technology is needed to improve the accuracy of doctors' diagnosis. Today's computer‐based medically assisted diagnostic technologies suffer from the inability to fully simulate physical attributes and environmental factors, the large computational resources required for high‐precision models, the need for professional training for user operation, and the limited intuition for innovative design. For improving diagnostic efficiency, this study designs a medical Question answering intelligent interaction system in view of artificial intelligence algorithms. The system is constructed with an active interactive intelligent Q&A model consisting of a medical reasoning module and a medical examination recommendation module. Then, it uses local Bayesian network algorithm as the foundation to establish an intelligent strategy optimization network. And it puts forward the answer selection model of medical Question answering in view of hierarchical interaction for natural language processing tasks in the medical context. The performance test results show that when the diagnostic end threshold of the medical reasoning module is 0.8, the shortest diagnostic path is 3.33. When the diagnostic threshold is 0.85, the maximum length of the diagnostic path is 4.66, and the maximum difference between the diagnostic paths is 1.33, which is basically not affected by the diagnostic end threshold. The local Bayesian network algorithm can reduce the impact of noise features and extract more valuable information. The accuracy of the multilevel interactive answer selection model on the Stanford Natural Language Inference dataset without using external resources reached 89.2%. The ablation test results show that the overall accuracy of the model is 89.64%. The visualization results of the attention weight distribution test between interaction layers show that under different levels of interaction, the attention distribution will undergo significant changes.

Label-aware debiased causal reasoning for Natural Language Inference

Recently, researchers have argued that the impressive performance of Natural Language Inference (NLI) models is highly due to the spurious correlations existing in training data, which makes models vulnerable and poorly generalized. Some work has made preliminary debiased attempts by developing data-driven interventions or model-level debiased learning. Despite the progress, existing debiased methods either suffered from the high cost of data annotation processing, or required elaborate design to identify biased factors. By conducting detailed investigations and data analysis, we argue that label information can provide meaningful guidance to identify these spurious correlations in training data, which has not been paid enough attention. Thus, we design a novel Label-aware Debiased Causal Reasoning Network (LDCRN). Specifically, according to the data analysis, we first build a causal graph to describe causal relations and spurious correlations in NLI. Then, we employ an NLI model (e.g., RoBERTa) to calculate total causal effect of input sentences to labels. Meanwhile, we design a novel label-aware biased module to model spurious correlations and calculate their causal effect in a fine-grained manner. The debiasing process is realized by subtracting this causal effect from total causal effect. Finally, extensive experiments over two well-known NLI datasets and multiple human-annotated challenging test sets are conducted to prove the superiority of LDCRN. Moreover, we have developed novel challenging test sets based on MultiNLI to facilitate the community.

Capturing the Varieties of Natural Language Inference: A Systematic Survey of Existing Datasets and Two Novel Benchmarks

AbstractTransformer-based Pre-Trained Language Models currently dominate the field of Natural Language Inference (NLI). We first survey existing NLI datasets, and we systematize them according to the different kinds of logical inferences that are being distinguished. This shows two gaps in the current dataset landscape, which we propose to address with one dataset that has been developed in argumentative writing research as well as a new one building on syllogistic logic. Throughout, we also explore the promises of ChatGPT. Our results show that our new datasets do pose a challenge to existing methods and models, including ChatGPT, and that tackling this challenge via fine-tuning yields only partly satisfactory results.

Textual entailment classification using syntactic structures and semantic relations

In this paper, we propose a method exploiting syntactic structure, semantic relations and word embeddings for recognizing textual entailment. The sentence pairs are analyzed using their syntactic structure and categorization of sentences in active voice, sentences in passive voice and sentences holding copular relations. The main syntactic relations such as subject, verb and object are extracted and lemmatized using a lemmatization algorithm based on parts-of-speech. The subject-to-subject, verb-to-verb and object-to-object similarity is identified using enhanced Wordnet semantic relations. Further similarity is analyzed using modifier relation, number relation, nominal modifier relation, compound relation, conjunction relation and negative relation. The experimental evaluation of the method on Stanford Natural Language Inference dataset shows that the accuracy of the method is 1.4% more when compared to the state-of-the-art zero shot domain adaptation methods.

Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialogues

Maintaining engagement and consistency is particularly important in dialogue systems. Existing works have improved the performance of dialogue systems by intentionally learning interlocutor personas with sophisticated network structures. One issue with this approach is that it requires more personal corpora with annotations. Additionally, these models typically perform the next utterance prediction to generate a response but neglect the discourse coherence in the entire conversation. To address these issues, this study proposes a method of learning to memorize entailment and discourse relations for persona-consistent dialogue tasks. Entailment text pairs in natural language inference dataset were applied to learn latent entailment relations as external memories by premise-to-hypothesis generation task. Furthermore, an internal memory with a similar architecture was applied to the discourse information in the dialogue. Placing orthogonality restrictions on these two memory spaces ensures that the latent entailment relations remain dialogue-independent. Both memories collaborate to obtain entailment and discourse representation for the generation, allowing a deeper understanding of both consistency and coherence. Experiments on two large public datasets, PersonaChat and DSTC7-AVSD, demonstrated the effectiveness of the proposed method. Both automatic and human evaluations indicate that the proposed model outperforms several strong baselines in terms of both persona consistency and response coherence. Our source code is availabled at https://github.com/Chenrj233/LMEDR.

A resampling-based method to evaluate NLI models

Abstract The recent progress of deep learning techniques has produced models capable of achieving high scores on traditional Natural Language Inference (NLI) datasets. To understand the generalization limits of these powerful models, an increasing number of adversarial evaluation schemes have appeared. These works use a similar evaluation method: they construct a new NLI test set based on sentences with known logic and semantic properties (the adversarial set), train a model on a benchmark NLI dataset, and evaluate it in the new set. Poor performance on the adversarial set is identified as a model limitation. The problem with this evaluation procedure is that it may only indicate a sampling problem. A machine learning model can perform poorly on a new test set because the text patterns presented in the adversarial set are not well represented in the training sample. To address this problem, we present a new evaluation method, the Invariance under Equivalence test (IE test). The IE test trains a model with sufficient adversarial examples and checks the model’s performance on two equivalent datasets. As a case study, we apply the IE test to the state-of-the-art NLI models using synonym substitution as the form of adversarial examples. The experiment shows that, despite their high predictive power, these models usually produce different inference outputs for equivalent inputs, and, more importantly, this deficiency cannot be solved by adding adversarial observations in the training data.

Curing the SICK and Other NLI Maladies

AbstractAgainst the backdrop of the ever-improving Natural Language Inference (NLI) models, recent efforts have focused on the suitability of the current NLI datasets and on the feasibility of the NLI task as it is currently approached. Many of the recent studies have exposed the inherent human disagreements of the inference task and have proposed a shift from categorical labels to human subjective probability assessments, capturing human uncertainty. In this work, we show how neither the current task formulation nor the proposed uncertainty gradient are entirely suitable for solving the NLI challenges. Instead, we propose an ordered sense space annotation, which distinguishes between logical and common-sense inference. One end of the space captures non-sensical inferences, while the other end represents strictly logical scenarios. In the middle of the space, we find a continuum of common-sense, namely, the subjective and graded opinion of a “person on the street.” To arrive at the proposed annotation scheme, we perform a careful investigation of the SICK corpus and we create a taxonomy of annotation issues and guidelines. We re-annotate the corpus with the proposed annotation scheme, utilizing four symbolic inference systems, and then perform a thorough evaluation of the scheme by fine-tuning and testing commonly used pre-trained language models on the re-annotated SICK within various settings. We also pioneer a crowd annotation of a small portion of the MultiNLI corpus, showcasing that it is possible to adapt our scheme for annotation by non-experts on another NLI corpus. Our work shows the efficiency and benefits of the proposed mechanism and opens the way for a careful NLI task refinement.

Evaluating Deep Learning Techniques for Natural Language Inference

Natural language inference (NLI) is one of the most important natural language understanding (NLU) tasks. NLI expresses the ability to infer information during spoken or written communication. The NLI task concerns the determination of the entailment relation of a pair of sentences, called the premise and hypothesis. If the premise entails the hypothesis, the pair is labeled as an “entailment”. If the hypothesis contradicts the premise, the pair is labeled a “contradiction”, and if there is not enough information to infer a relationship, the pair is labeled as “neutral”. In this paper, we present experimentation results of using modern deep learning (DL) models, such as the pre-trained transformer BERT, as well as additional models that relay on LSTM networks, for the NLI task. We compare five DL models (and variations of them) on eight widely used NLI datasets. We trained and fine-tuned the hyperparameters for each model to achieve the best performance for each dataset, where we achieved some state-of-the-art results. Next, we examined the inference ability of the models on the BreakingNLI dataset, which evaluates the model’s ability to recognize lexical inferences. Finally, we tested the generalization power of our models across all the NLI datasets. The results of the study are quite interesting. In the first part of our experimentation, the results indicate the performance advantage of the pre-trained transformers BERT, RoBERTa, and ALBERT over other deep learning models. This became more evident when they were tested on the BreakingNLI dataset. We also see a pattern of improved performance when the larger models are used. However, ALBERT, given that it has 18 times fewer parameters, achieved quite remarkable performance.

AmericasNLI: Machine translation and natural language inference systems for Indigenous languages of the Americas.

Little attention has been paid to the development of human language technology for truly low-resource languages-i.e., languages with limited amounts of digitally available text data, such as Indigenous languages. However, it has been shown that pretrained multilingual models are able to perform crosslingual transfer in a zero-shot setting even for low-resource languages which are unseen during pretraining. Yet, prior work evaluating performance on unseen languages has largely been limited to shallow token-level tasks. It remains unclear if zero-shot learning of deeper semantic tasks is possible for unseen languages. To explore this question, we present AmericasNLI, a natural language inference dataset covering 10 Indigenous languages of the Americas. We conduct experiments with pretrained models, exploring zero-shot learning in combination with model adaptation. Furthermore, as AmericasNLI is a multiway parallel dataset, we use it to benchmark the performance of different machine translation models for those languages. Finally, using a standard transformer model, we explore translation-based approaches for natural language inference. We find that the zero-shot performance of pretrained models without adaptation is poor for all languages in AmericasNLI, but model adaptation via continued pretraining results in improvements. All machine translation models are rather weak, but, surprisingly, translation-based approaches to natural language inference outperform all other models on that task.

Network based on the synergy of knowledge and context for natural language inference

The goal of natural language inference (NLI) is to judge the logical relationship between sentence pairs, including entailment, contradiction, and neutral. At present, many researchers have shown that the introduction of external knowledge is helpful for NLI. However, the existing models are not effective enough for the utilization of knowledge information and not sufficient for the exploration of context information. In this paper, we propose a multi-branch network based on the synergy of knowledge and context. It applies two kinds of information synergy in the way that both knowledge information and context information promote performance improvement while modeling their separately in independent branches. In the context branch, we present a multi-level and dynamic assisted attention to construct sufficient interaction between sentence pairs. In the knowledge branch, we design a Knowledge-based Graph Attention Network (K-GAT) to capture the structural information of the knowledge, and the attention mechanism is used for knowledge interaction. In addition, for strengthening the relation between sentence pairs, we provide a relation branch to capture the context and knowledge relations of sentence pairs. In order to avoid introducing redundant external knowledge as much as possible, we adopt the introduction of five selected semantic dependencies based knowledge types. Experiments show that our model achieves strong competitive results on all three popular NLI datasets.

Building a Vietnamese Dataset for Natural Language Inference Models.

Natural language inference models are essential resources for many natural language understanding applications. These models are possibly built by training or fine-tuning using deep neural network architectures for state-of-the-art results. That means high-quality annotated datasets are essential for building state-of-the-art models. Therefore, we propose a method to build a Vietnamese dataset for training Vietnamese inference models which work on native Vietnamese texts. Our approach aims at two issues: removing cue marks and ensuring the writing style of Vietnamese texts. If a dataset contains cue marks, the trained models will identify the relationship between a premise and a hypothesis without semantic computation. For evaluation, we fine-tuned a BERT model, viNLI, on our dataset and compared it to a BERT model, viXNLI, which was fine-tuned on XNLI dataset. The viNLI model has an accuracy of 94.79%, while the viXNLI model has an accuracy of 64.04% when testing on our Vietnamese test set. In addition, we also conducted an answer selection experiment with these two models in which the P@1 of viNLI and of viXNLI are 0.4949 and 0.4044, respectively. That means our method can be used to build a high-quality Vietnamese natural language inference dataset.

Graph Enhanced Representation and Reasoning Model for Tabular Fact Verification

Tabular fact verification is a challenging task that requires obtaining relevant evidence from the table and utilizing them to verify a given claim. The main difficulty in tabular fact verification is that traditional language models cannot capture the underlying information carried in tabular data. To solve this problem, we propose GraERR, a Graph Enhanced Representation and Reasoning Model for Tabular Fact Verification. It consists of two modules: a data initial representation module based on the DeBERTa model and a graph-augmented representation and inference module. The former improves the inference ability of the model with a DeBERTa model pre-trained on the natural language inference dataset. The graph enhancement module enhances the model’s ability to learn table representations by effectively integrating textual and structural information in tables. The best results obtained experimentally on the InfoTabs dataset demonstrate the effectiveness of the GECMT model on the tabular fact verification task.

Supervising Model Attention with Human Explanations for Robust Natural Language Inference

Natural Language Inference (NLI) models are known to learn from biases and artefacts within their training data, impacting how well they generalise to other unseen datasets. Existing de-biasing approaches focus on preventing the models from learning these biases, which can result in restrictive models and lower performance. We instead investigate teaching the model how a human would approach the NLI task, in order to learn features that will generalise better to previously unseen examples. Using natural language explanations, we supervise the model’s attention weights to encourage more attention to be paid to the words present in the explanations, significantly improving model performance. Our experiments show that the in-distribution improvements of this method are also accompanied by out-of-distribution improvements, with the supervised models learning from features that generalise better to other NLI datasets. Analysis of the model indicates that human explanations encourage increased attention on the important words, with more attention paid to words in the premise and less attention paid to punctuation and stopwords.

SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

Abstract In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SummaC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.

Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts

Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like checking the factual correctness of summaries.

Natural language inference for Malayalam language using language agnostic sentence representation

Natural language inference (NLI) is an essential subtask in many natural language processing applications. It is a directional relationship from premise to hypothesis. A pair of texts is defined as entailed if a text infers its meaning from the other text. The NLI is also known as textual entailment recognition, and it recognizes entailed and contradictory sentences in various NLP systems like Question Answering, Summarization and Information retrieval systems. This paper describes the NLI problem attempted for a low resource Indian language Malayalam, the regional language of Kerala. More than 30 million people speak this language. The paper is about the Malayalam NLI dataset, named MaNLI dataset, and its application of NLI in Malayalam language using different models, namely Doc2Vec (paragraph vector), fastText, BERT (Bidirectional Encoder Representation from Transformers), and LASER (Language Agnostic Sentence Representation). Our work attempts NLI in two ways, as binary classification and as multiclass classification. For both the classifications, LASER outperformed the other techniques. For multiclass classification, NLI using LASER based sentence embedding technique outperformed the other techniques by a significant margin of 12% accuracy. There was also an accuracy improvement of 9% for LASER based NLI system for binary classification over the other techniques.

Is your document novel? Let attention guide you. An attention-based model for document-level novelty detection

AbstractDetecting, whether a document contains sufficient new information to be deemed as novel, is of immense significance in this age of data duplication. Existing techniques for document-level novelty detection mostly perform at the lexical level and are unable to address the semantic-level redundancy. These techniques usually rely on handcrafted features extracted from the documents in a rule-based or traditional feature-based machine learning setup. Here, we present an effective approach based on neural attention mechanism to detect document-level novelty without any manual feature engineering. We contend that the simple alignment of texts between the source and target document(s) could identify the state of novelty of a target document. Our deep neural architecture elicits inference knowledge from a large-scale natural language inference dataset, which proves crucial to the novelty detection task. Our approach is effective and outperforms the standard baselines and recent work on document-level novelty detection by a margin of $\sim$ 3% in terms of accuracy.

SA-NLI: A Supervised Attention based framework for Natural Language Inference

Natural Language Inference (NLI) aims to determine the relationship of a pair of sentences. As a critical component of NLI models, attention mechanism has been proven to be effective in the representation and interaction of sentences. However, the attention methods adopted in the existing NLI models are non-parametric or trained inside the model without explicit supervision, and the attention results are poorly explained. In this paper, we propose a Supervised Attention based Natural Language Inference (SA-NLI) framework to solve this problem. In our framework, the intra attention module is trained to focus on syntactically related tokens, while the inter attention module is constrained to capture alignment between sentences. Moreover, the supervised training of intra attention module and inter attention module are unified with the training of the NLI model by multi-task learning and transfer learning, respectively. We conduct extensive experiments on multiple NLI datasets, and the results demonstrate the effectiveness of our supervised attention based method. Further visual analysis validates the interpretability of attention results, and the extended experimental results indicate the generalization of our SA-NLI framework.