Abstract

Sentence matching is a key issue in natural language inference and paraphrase identification. Despite the recent progress on multi-layered neural network with cross sentence attention, one sentence learns attention to the intermediate representations of another sentence, which are propagated from preceding layers and therefore are uncertain and unstable for matching, particularly at the risk of error propagation. In this paper, we present an original semantics-oriented attention and deep fusion network (OSOA-DFN) for sentence matching. Unlike existing models, each attention layer of OSOA-DFN is oriented to the original semantic representation of another sentence, which captures the relevant information from a fixed matching target. The multiple attention layers allow one sentence to repeatedly read the important information of another sentence for better matching. We then additionally design deep fusion to propagate the attention information at each matching layer. At last, we introduce a self-attention mechanism to capture global context to enhance attention-aware representation within each sentence. Experiment results on three sentence matching benchmark datasets SNLI, SciTail and Quora show that OSOA-DFN has the ability to model sentence matching more precisely.

Highlights

  • Natural language sentence matching is a key technique of comparing two sentences and identifying the semantic relationship between them, which is usually viewed as a classification problem (Wang et al, 2017)

  • Match(·) is mainly focused by researches and some effective frameworks are proposed (Rocktaschel et al, 2015; Wang et al, 2017; Duan et al, 2018). We focus on this layer, and propose an original semanticsoriented attention and deep fusion network

  • We evaluate our model on natural language inference and paraphrase identification tasks with three datasets: the Stanford Natural Language Inference (SNLI) dataset (Bowman et al, 2015), the SciTail dataset (Khot et al, 2018), and the Quora Questions Pairs dataset (Quora)

Read more

Summary

Introduction

Natural language sentence matching is a key technique of comparing two sentences and identifying the semantic relationship between them, which is usually viewed as a classification problem (Wang et al, 2017). For the second type of framework, attention mechanism is introduced to model word-level interaction between two sentences and a higher accuracy is achieved (Rocktaschel et al, 2015; Parikh et al, 2016; Wang et al, 2017). In order to address these problems, we propose an original semantics-oriented attention and deep fusion network (OSOA-DFN) for sentence matching. We pay attention to the original semantic representations for cross sentence interaction and the matching target of attention for a certain sentence is ensured to be fixed in spite of multiple layers. We design a deep fusion in addition to usual fusion to augment the propagation of attention information for matching, and introduce a self-attention mechanism at the last to capture global context to enhance attentionaware representation within each sentence. We evaluate our model on three challenging datasets and show that the proposed model has the ability to model sentence matching more precisely and significantly improves the performance

General Neural Attention-Based Model for Sentence Matching
Input Encoding Layer
Attention-Based Matching Layer
Prediction Layer
Original Semantics-Oriented Attention and Deep Fusion Network
Original Semantics-Oriented Cross Sentence Attention
Deep Fusion
Self-Attention Mechanism
Training
Dataset
Implementation Details
Ensemble
Comparison on Natural Language Inference
Comparison on Paraphrase Identification
Effect of Original Semantics-Oriented Cross Sentence Attention
Effect of Deep Fusion and Self-Attention Mechanism
What is Learned by Attention ?
Related Works
Findings
Conclusions and Future Work
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call