Abstract

We describe the Uppsala NLP submission to SemEval-2021 Task 2 on multilingual and cross-lingual word-in-context disambiguation. We explore the usefulness of three pre-trained multilingual language models, XLM-RoBERTa (XLMR), Multilingual BERT (mBERT) and multilingual distilled BERT (mDistilBERT). We compare these three models in two setups, fine-tuning and as feature extractors. In the second case we also experiment with using dependency-based information. We find that fine-tuning is better than feature extraction. XLMR performs better than mBERT in the cross-lingual setting both with fine-tuning and feature extraction, whereas these two models give a similar performance in the multilingual setting. mDistilBERT performs poorly with fine-tuning but gives similar results to the other models when used as a feature extractor. We submitted our two best systems, fine-tuned with XLMR and mBERT.

Highlights

  • SemEval-2021 Task 2: Multilingual and Crosslingual Word-in-Context Disambiguation (MCLWiC) (Martelli et al, 2021) is an extension from WiC (Pilehvar and Camacho-Collados, 2019), a shared task at the IJCAI-19 SemDeep workshop (SemDeep-5)

  • XLMR gives the best results for all cross-lingual language pairs, with an improvement over Multilingual BERT (mBERT) of 4.1– 10.5 percentage points

  • We found that fine-tuning the language models is preferable to using them as feature extractors either for an multi-layer perceptron (MLP) or for logistic regression

Read more

Summary

Introduction

SemEval-2021 Task 2: Multilingual and Crosslingual Word-in-Context Disambiguation (MCLWiC) (Martelli et al, 2021) is an extension from WiC (Pilehvar and Camacho-Collados, 2019), a shared task at the IJCAI-19 SemDeep workshop (SemDeep-5). WiC was proposed as a benchmark to evaluate context-sensitive word representations. The WiC dataset consists of a list of English sentence-pairs. Each sentence-pair has a target word, and the task is to determine whether the target word is used in the same meaning or different meanings in the two sentences, as a binary classification task. MCL-WiC extends WiC to multilingual and cross-lingual datasets, and covers 5. Example The cat chases after the mouse. La souris mange le fromage. (‘The mouse eats the cheese’)

Methods
Results
Conclusion

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.