Abstract

Neural natural language generation (NLG) and understanding (NLU) models are data-hungry and require massive amounts of annotated data to be competitive. Recent frameworks address this bottleneck with generative models that synthesize weak labels at scale, where a small amount of training labels are expert-curated and the rest of the data is automatically annotated. We follow that approach, by automatically constructing a large-scale weakly-labeled data with a fine-tuned GPT-2, and employ a semi-supervised framework to jointly train the NLG and NLU models. The proposed framework adapts the parameter updates to the models according to the estimated label-quality. On both the E2E and Weather benchmarks, we show that this weakly supervised training paradigm is an effective approach under low resource scenarios and outperforming benchmark systems on both datasets when 100% of training data is used.

Highlights

  • Natural language generation (NLG) is the task that transforms meaning representations (MR) into natural language descriptions (Reiter and Dale, 2000; Barzilay and Lapata, 2005); while natural language understanding (NLU) is the opposite process where text is converted into MR (Zhang and Wang, 2016)

  • Work on semi-supervised learning considers settings with some labeled data and a much larger set of unlabeled data, and leverages both labeled the unlabeled data as in machine translation (Artetxe et al, 2017; Lample et al, 2017), data-to-text generation (Schmitt and Schutze, 2019; Qader et al, 2019) or more relevantly the joint learning framework for training NLU and natural language generation (NLG) (Tseng et al, 2020; Su et al, 2020). These approaches all assume that a large collection of text is available, which is an unrealistic assumption for the task due to the need for expert curation. We show that both NLU and NLG models can benefit from (1) automatically labeling MR with text, and (2) by semi-supervisedly learning from these samples while accounting for their qualities

  • We analyze the strength of the weak annotator, and the effect of the qualityweighted weak supervision, before concluding with the analysis of dual mutual information

Read more

Summary

Introduction

Natural language generation (NLG) is the task that transforms meaning representations (MR) into natural language descriptions (Reiter and Dale, 2000; Barzilay and Lapata, 2005); while natural language understanding (NLU) is the opposite process where text is converted into MR (Zhang and Wang, 2016) These two processes can constrain each other – recent exploration of the duality of neural natural language generation (NLG) and understanding (NLU) has led to successful semi-supervised learning techniques where both labeled and unlabeled data can be used for training (Su et al, 2020; Tseng et al, 2020; Schmitt and Schutze, 2019; Qader et al, 2019; Su et al, 2020). Labeled data and large unlabeled data can be utilized in semi-supervised learning (Lample et al, 2017; Tseng et al, 2020), as a way to jointly improve both NLU and NLG models

Methods
Results
Conclusion
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.