Filipino Native Language Identification using Markov Chain Model and Maximum Likelihood Decision Rule

Ria Ambrocio Sagum

doi:10.17762/turcomat.v12i3.2206

Abstract

The study developed a tool for identification of a Filipino Native Language given a textual data. The Filipino Language identified were Cebuano, Kapampangan and Pangasinan. It used Markov Chain Model for language modeling using bag of words (a total of 35,144 words for Cebuano, 14752 for Kapampangan, and 13969 of Pangasinan) from each language and maximum likelihood decision rule for the identification of the native language. The obtained model implementing Markov model, was applied in one hundred fifty text files with minimum length of ten words and maximum length of fifty words. The result of the evaluation shows the system’s accuracy of 86.25% and an F-Score of 90.55%.

Highlights

The Philippines is an archipelago that consists of more than 7000 islands [1]
Language identification can be used for Filipino dialects to for most of the national language of every country already exists, but as said, the Filipino language for example is characterized by variety
This study aims to create a native language identification tool that recognizes 3 of the 8 major dialects or languages in the Philippines

Summary

Introduction

The Philippines is an archipelago that consists of more than 7000 islands [1]. A factor why the Philippines is considered as multilingual country or country with different languages resulting in a different interpretation amongst the meaning of the word within the country. The variation may go largely unnoticed or overlooked Once this happen this will be the beginning of the misunderstanding of people as sometimes a word may have different meaning depending on the dialect. Language identification is one of the pre-processing units in natural language processing that can be used to do this task This identification can be done through statistical computing and works by identifying patterns [2]. Language identification can be used for Filipino dialects to for most of the national language of every country already exists, but as said, the Filipino language for example is characterized by variety. These variations may affect and cause failure with succeeding pre-processing units of NLP. Knowing the dialects of the Filipino language, this may work, but the fact that these dialects are somehow based from Tagalog, the structure of words may still show potential similarities

Objectives

Methods

Findings

Discussion

Conclusion