Abstract
Recent work on word ordering has argued that syntactic structure is important, or even required, for effectively recovering the order of a sentence. We find that, in fact, an n-gram language model with a simple heuristic gives strong results on this task. Furthermore, we show that a long short-term memory (LSTM) language model is even more effective at recovering order, with our basic model outperforming a state-of-the-art syntactic model by 11.5 BLEU points. Additional data and larger beams yield further gains, at the expense of training and search time.
Highlights
We address the task of recovering the original word order of a shuffled sentence, referred to as bag generation (Brown et al, 1990), shake-and-bake generation (Brew, 1992), or more recently, linearization, as standardized in a recent line of research as a method useful for isolating the performance of text-to-text generation models (Zhang and Clark, 2011; Liu et al, 2015; Liu and Zhang, 2015; Zhang and Clark, 2015)
Selective restrictions, subcategorization, and discourse considerations are among the many factors which join together to fix the order in which words occur. . . [T]here is an abstract structure which underlies the surface strings and it is this structure which provides a more insightful basis for understanding the constraints on word order. . . . It is, an interesting question to ask whether a network can learn any aspects of that underlying abstract structure (Elman, 1990)
We find that language models are in general effective for linearization relative to existing syntactic approaches, with long short-term memory (LSTM) in particular outperforming the state-of-the-art by 11.5 BLEU points, with further gains observed when training with additional text and decoding with larger beams
Summary
The predominant argument of the more recent works is that jointly recovering explicit syntactic structure is crucial for determining the correct word order of the original sentence. As such, these methods either generate or rely on given parse structure to reproduce the order. Elman judged the capacity of early recurrent neural networks via, in part, the network’s ability to predict word order in simple sentences. He notes, The order of words in sentences reflects a number of constraints.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.