Evaluating the predication model of metaphor comprehension: Using word2vec to model best/worst quality judgments of 622 novel metaphors.

Abstract

In this paper our goal is to undertake a systematic assessment of the first, most widely known, and simplest computational model of metaphor comprehension, the predication model developed by Kintsch (Cognitive Science, 25(2), 173-202, 2000). 622 metaphors of the form "x is a y" were selected from a much larger set generated randomly. The metaphors were judged for quality using best/worst judgments, which asks judges to pick the best and worst metaphor from among four presented metaphors. The metaphors and their judgments have been publicly released. We modeled the judgments by extending Kintsch's predication model (2000) by systematically walking through the parameter space of that model. Our model successfully differentiated metaphors rated as good (> 1.5z) from metaphors rated as bad (< -1.5z; Cohen's d = 0.72) and was able to successfully classify good metaphors with an accuracy of 82.9%. However, it achieved a true negative rate below chance at 36.3% and had a resultantly low kappa of 0.037. The model could not distinguish unselected random metaphors from those selected by humans as having metaphorical potential. In a follow-up study we showed that the model's quality estimates reliably predict metaphor decision times, with better metaphors being judged more quickly than worse metaphors.