Language Models Use Monotonicity to Assess NPI Licensing

Jaap Jumelet,Jakub Szymanik,Shane Steinert-Threlkeld,Milica Denic,Dieuwke Hupkes

doi:10.18653/v1/2021.findings-acl.439

Abstract

We investigate the semantic knowledge of language models (LMs), focusing on (1) whether these LMs create categories of linguistic environments based on their semantic monotonicity properties, and (2) whether these categories play a similar role in LMs as in human language understanding, using negative polarity item licensing as a case study. We introduce a series of experiments consisting of probing with diagnostic classifiers (DCs), linguistic acceptability tasks, as well as a novel DC ranking method that tightly connects the probing results to the inner workings of the LM. By applying our experimental pipeline to LMs trained on various filtered corpora, we are able to gain stronger insights into the semantic generalizations that are acquired by these models.

Highlights

Introduction has confirmed this connection betweenNPIs and monotonicity: humans judge NPIs acceptable in Neural language models (LMs) have become pow- a linguistic environment if they consider that enerful approximators of human language, making it vironment to be downward monotone (Chemla increasingly important to understand the features et al, 2011)
To investigate monotonicity we focus on negatrained on various filtered corpora, we are able tive polarity items (NPIs): a class of expressions to gain stronger insights into the semantic generalizations that are acquired by these models.1 such as any or ever that are solely acceptable in downward monotone environments (Fauconnier, 1975; Ladusaw, 1979)
To create a corpus of monotonicity sentences for training and testing the diagnostic classifiers (DCs), we leverage the corpus of Warstadt et al (2019), selecting all downward monotone (DM) and upward monotone (UM) sentences to build up a balanced corpus of these categories

Summary

Introduction

NPIs and monotonicity: humans judge NPIs acceptable in Neural language models (LMs) have become pow- a linguistic environment if they consider that enerful approximators of human language, making it vironment to be downward monotone (Chemla increasingly important to understand the features et al, 2011). Previous research has established that and mechanisms underlying their behavior (Linzen LMs are relatively successful in processing NPIs et al, 2018, 2019). Such work has focused primarily on syntactic properties, while fewer studies have been done on what kind of formal semantic features are encoded by language. RQ1 Do language models encode the monotonicity properties of linguistic environments?. We focus explicitly on what LMs learn about a semantic property of sentences, and in what ways their knowledge reflects wellknown features of human language processing. As the topic of our studies, we consider monotonicity, a semantic property of linguistic envi-

Objectives

Methods

Results

Conclusion