Decoding Methods for Neural Narrative Generation

Alexandra DeLucia,João Sedoc,Aaron Mueller,Xiang Lisa Li

doi:10.18653/v1/2021.gem-1.16

Abstract

Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. The task is similar to neural response generation for chatbots; however, innovations in response generation are often not applied to narrative generation, despite the similarity between these tasks. We aim to bridge this gap by applying and evaluating advances in decoding methods for neural response generation to neural narrative generation. In particular, we employ GPT-2 and perform ablations across nucleus sampling thresholds and diverse decoding hyperparameters—specifically, maximum mutual information—analyzing results over multiple criteria with automatic and human evaluation. We find that (1) nucleus sampling is generally best with thresholds between 0.7 and 0.9; (2) a maximum mutual information objective can improve the quality of generated stories; and (3) established automatic metrics do not correlate well with human judgments of narrative quality on any qualitative metric.

Highlights

Narrative generation is the task of generating a creative response given an input prompt
Recent advances in decoding methods for response generation—notably, the application of the maximum mutual information (MMI) objective (Li et al, 2016a)—have resulted in more interesting dialog according to human evaluators (Zhang et al, 2020b); this has not been applied to narrative generation
See et al (2019) extend Fan et al (2018), but use GPT-2 small and perform a top-k decoding parameter sweep. We focus on this open-ended narrative generation task in our investigation, but primarily focus on GPT-2 Medium and on the effect of nucleus sampling thresholds

Summary

Introduction

Narrative generation (or story generation) is the task of generating a creative response given an input prompt. This output can be a story closure, a paragraph, or a structured story with multiple paragraphs. This input and output setup is similar to the response generation task of chatbots, as both tasks convert some variable-length sequential input from a user to an automatically generated variablelength sequential output. Recent advances in decoding methods for response generation—notably, the application of the maximum mutual information (MMI) objective (Li et al, 2016a)—have resulted in more interesting dialog according to human evaluators (Zhang et al, 2020b); this has not been applied to narrative generation. The MMI objective has been confined to short-form and less openended generation tasks far

Objectives

Methods

Results

Conclusion