Abstract

Named Entity Recognition (NER) is a major task in the field of Natural Language Processing (NLP), and also is a sub-task of Information Extraction. The challenge of NER for tweets lie in the insufficient information available in a tweet. There has been a significant amount of work done related to entity extraction, but only for resource rich languages and domains such as newswire. Entity extraction is, in general, a challenging task for such an informal text, and code-mixed text further complicates the process with it’s unstructured and incomplete information. We propose experiments with different machine learning classification algorithms with word, character and lexical features. The algorithms we experimented with are Decision tree, Long Short-Term Memory (LSTM), and Conditional Random Field (CRF). In this paper, we present a corpus for NER in Hindi-English Code-Mixed along with extensive experiments on our machine learning models which achieved the best f1-score of 0.95 with both CRF and LSTM.

Highlights

  • Multilingual speakers often switch back and forth between languages when speaking or writing, mostly in informal settings

  • Bali et al performed analysis of data from Facebook posts generated by English-Hindi bilingual users

  • Sharma et al addressed the problem of shallow parsing of Hindi-English code-mixed social media text and developed a system for HindiEnglish code-mixed text that can identify the language of the words, normalize them to their standard forms, assign them their POS tag and segment into chunks

Read more

Summary

Introduction

Multilingual speakers often switch back and forth between languages when speaking or writing, mostly in informal settings. Code-mixing refers to the use of linguistic units from different languages in a single utterance or sentence, whereas codeswitching refers to the co-occurrence of speech extracts belonging to two different grammatical systems Gumperz As both phenomena are frequently observed on social media platforms in similar contexts, we use only the code-mixing scenario in this work. Vyas et al formalized the problem, created a POS tag annotated Hindi-English code-mixed corpus and reported the challenges and problems in the HindiEnglish code-mixed text. They performed experiments on language identification, transliteration, normalization and POS tagging of the Dataset. Barman et al addressed the problem of language identification on BengaliHindi-English Facebook comments

Methods
Results
Conclusion
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.