Arabic NER Evaluation: Pre-Trained Models via Contrastive Learning vs. LLM Few-Shot Prompting

Abstract

Developing Natural Language Processing (NLP) tools for the Arabic language and its dialects is very challenging. Named Entity Recognition (NER) is one of these challenges, which serves as the core component in many NLP systems such as information extraction, question answering, machine translation and knowledge graph building. This paper sheds light on applying diferent approaches for Arabic NER (Flat and Nested) using a large and rich Arabic NER corpus, Wojood dataset, which consists of about 550K tokens annotated with 21 entity types. First, we apply the Wojood base model, AraBERTv2, along with various other Arabic BERT models such as MARBERTv2, CaMelBert, mBert, ..etc. Next, we utilize the Bi-Encoder Contrastive Learning (CL) approach, a framework developed by Microsoft, which maps candidate text spans and entity types into the same vector representation space. The primary challenge in this approach is distinguishing non-entity spans from entity mentions. This approach could achieve F1 score 91.25% for Flat and 91.40% for Nested NER. Additionally, for evaluating the predicted NER, we employ Few-Shot prompting on LLaMA, and GPT-3.5 using refined prompt-based strategy. Our findings reveal that LLaMA outperforms GPT3.5.