Abstract
The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering inference speed, making state-of-the-art models cumbersome to deploy in real-world, time-sensitive settings. We develop a compression technique for autoregressive models that is driven by an imitation learning perspective on knowledge distillation. The algorithm is designed to address the exposure bias problem. On prototypical language generation tasks such as translation and summarization, our method consistently outperforms other distillation algorithms, such as sequence-level knowledge distillation. Student models trained with our method attain 1.4 to 4.8 BLEU/ROUGE points higher than those trained from scratch, while increasing inference speed by up to 14 times in comparison to the teacher model.
Highlights
IntroductionDue to the sequential nature of text generation, they are often the tool of choice for tackling sequence-to-sequence problems such as translation (Sutskever et al, 2014), summarization (Rush et al, 2015), and dialogue (Eric and Manning, 2017)
Autoregressive models are ubiquitous in natural language processing
Two recent trends have made autoregressive models cumbersome to deploy in real-world, natural language generation (NLG) applications
Summary
Due to the sequential nature of text generation, they are often the tool of choice for tackling sequence-to-sequence problems such as translation (Sutskever et al, 2014), summarization (Rush et al, 2015), and dialogue (Eric and Manning, 2017) They form the backbone of several successful generative pre-training architectures (Howard and Ruder, 2018; Peters et al, 2018; Radford et al, 2019; Dai et al, 2019). The joint distribution over y may itself be conditional on some related source feature x ∈ X (e.g. translation, summarization) or not (e.g. language modeling) Since the former case can generalize the latter by letting X = ∅, we will specify the presence of x in the rest of the paper. The training objective can be expressed as
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
