Adversarial Datasets for NLI Tasks: the Case of the Chinese Causative-Passive Homonymy

Abstract

We present the first Chinese adversarial natural language inference (NLI) evaluation set (CANLI) to probe the limits of pretrained language models (PLMs). CANLI is based on the linguistic phenomenon of causative- passive homonymy (CPH). The disambiguation of CPH is easy for humans, but difficult for machines, as the causative and the passive are not distinguished by the sentences’ syntactic structure. We created the CANLI in two related settings: (i) CA: a syntactic heuristic setting and (ii) CB: a semantic role labeling setting. We find that state-of-the-art PLMs that are fine-tuned on existing large-scale Chinese NLI benchmark datasets perform poorly on CANLI. Further, we notice that the PLM’s performance on CA and CB is independent of each other, suggesting that the PLM has not learned the shared underlying linguistic knowledge CPH, but instead the respective spurious patterns. We conclude that there is considerable room for improvement in the NLI system.