Abstract
We present the first Chinese adversarial natural language inference (NLI) evaluation set (CANLI) to probe the limits of pretrained language models (PLMs). CANLI is based on the linguistic phenomenon of causative- passive homonymy (CPH). The disambiguation of CPH is easy for humans, but difficult for machines, as the causative and the passive are not distinguished by the sentences’ syntactic structure. We created the CANLI in two related settings: (i) CA: a syntactic heuristic setting and (ii) CB: a semantic role labeling setting. We find that state-of-the-art PLMs that are fine-tuned on existing large-scale Chinese NLI benchmark datasets perform poorly on CANLI. Further, we notice that the PLM’s performance on CA and CB is independent of each other, suggesting that the PLM has not learned the shared underlying linguistic knowledge CPH, but instead the respective spurious patterns. We conclude that there is considerable room for improvement in the NLI system.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.