How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis

Abstract

In this paper, we try to answer “How Pre-trained Language Models Capture Factual Knowledge” by quantitatively measuring and evaluating the word-level patterns that PLMs depend on to capture the factual words. We check the words that have three typical associations with the missing words: knowledge-dependent, positionally close, and highly co-occurred. Our analysis shows: (1) PLMs generate the missing factual words more by the positionally close and highly co-occurred words than the knowledge-dependent words; (2) the dependence on the knowledge-dependent words is more effective than the positionally close and highly co-occurred words. Accordingly, we conclude that the PLMs capture the factual knowledge ineffectively because depending on the inadequate associations.