Abstract

Semantic parsing considers the task of transducing natural language (NL) utterances into machine executable meaning representations (MRs). While neural network-based semantic parsers have achieved impressive improvements over previous methods, results are still far from perfect, and cursory manual inspection can easily identify obvious problems such as lack of adequacy or coherence of the generated MRs. This paper presents a simple approach to quickly iterate and improve the performance of an existing neural semantic parser by reranking an n-best list of predicted MRs, using features that are designed to fix observed problems with baseline models. We implement our reranker in a competitive neural semantic parser and test on four semantic parsing (GEO, ATIS) and Python code generation (Django, CoNaLa) tasks, improving the strong baseline parser by up to 5.7% absolute in BLEU (CoNaLa) and 2.9% in accuracy (Django), outperforming the best published neural parser results on all four datasets.

Highlights

  • Semantic parsing is the task of mapping a natural language utterance into machine executable meaning representations (e.g., Python code)

  • Recent years have witnessed a burgeoning of applying neural network architectures for semantic parsing, from sequence-to-sequence models (Jia and Liang, 2016; Ling et al, 2016; Liang et al, 2017; Suhr et al, 2018), to more complex parsing paradigms guided by the structured topologies of target meaning representations (Xiao et al (2016); Dong and Lapata (2016); Yin and Neubig (2017); Rabinovich et al (2017); Krishnamurthy et al (2017); Zhong et al (2017); Dong and Lapata (2018); Iyer et al (2018), inter alia)

  • We observe that around 41% of the failure cases of TRANX on a popular Python code generation task (DJANGO, Oda et al (2015)) are due to such inadequate predictions

Read more

Summary

Introduction

Semantic parsing is the task of mapping a natural language utterance into machine executable meaning representations (e.g., Python code). C 2019 Association for Computational Linguistics tivates us to investigate whether the performance of an existing neural parser can be potentially improved by reranking the n-best list of candidate MRs. In this paper, we propose a simple reranker powered mainly by two quality-measuring features of a candidate MR: (1) a generative reconstruction model, which tests the coherence and adequacy of an MR via the likelihood of reconstructing the original input utterance from the MR; and (2) a discriminative matching model, which directly captures the semantic coherence between utterances and MRs. We implement our reranker in a strong neural semantic parser and evaluate on both tasks of parsing NL to domain-specific logical form (GEO, ATIS) and general-purpose source code (DJANGO, CONALA). Our reranking approach improves upon this strong parser by up to 5.7% absolute in BLEU (CONALA) and 2.9% in accuracy (DJANGO), outperforming the best published neural parser results on all datasets

Reranking Model
Aggregation
Experiment
Results
Conclusion
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call