Abstract
Neural language models learn, to varying degrees of accuracy, the grammatical properties of natural languages. In this work, we investigate whether there are systematic sources of variation in the language models' accuracy. Focusing on subject-verb agreement and reflexive anaphora, we find that certain nouns are systematically understood better than others, an effect which is robust across grammatical tasks and different language models. Surprisingly, we find that across four orders of magnitude, corpus frequency is unrelated to a noun's performance on grammatical tasks. Finally, we find that a novel noun's grammatical properties can be few-shot learned from various types of training data. The results present a paradox: there should be less variation in grammatical performance than is actually observed.
Highlights
Neural language models (Howard and Ruder, 2018; Devlin et al, 2019; Dai et al, 2019; Yang et al, 2019; Radford et al, 2019) have achieved success in both text prediction and downstream tasks such as question-answering, text classification, and natural language inference
Previous work has investigated the linguistic representations of neural language models in several domains, and found varying evidence for how linguistically adequate these representations are (Lau et al, 2017; Marvin and Linzen, 2018; Goldberg, 2019; Futrell et al, 2019)
We focus on the variation in grammatical knowledge that potentially exists within a neural language model
Summary
Neural language models (Howard and Ruder, 2018; Devlin et al, 2019; Dai et al, 2019; Yang et al, 2019; Radford et al, 2019) have achieved success in both text prediction and downstream tasks such as question-answering, text classification, and natural language inference. This work has employed psycholinguistic methodology in order to elicit grammatical judgments from these models, inferring the models’ underlying representations from the patterns of judgments. We focus on the variation in grammatical knowledge that potentially exists within a neural language model. Just as in human psycholinguistic tasks, previous work on neural LMs has observed variability in grammatical judgments between different sentences; not all violations of a grammatical constraint are judged to be bad. It is not clear, whether there are systematic sources of variation in these judgments, and if so, what the sources are
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
