Abstract
This paper describes the design and creation of a monolingual parallel corpus for the Malay language written in Jawi.This paper proposes a new corpus called the National University of Malaysia Word Tokenization (NUWT) corpora To the best of our knowledge, currently, there is no sufficiently comprehensive, well-designed standard corpus that is annotated and made available for the public for the Jawi script corpora.This corpus contains the Jawi-specific Buckwalter character code and can be used to evaluate the performance of word tokenization tasks, as well as further language processing.The objective of this work is to conform and standardize the corpora between similar characters in Jawi.It consists of three subcorporas with documents from different genres. The gathering and processing steps, as well as the definition of several evaluation tasks regarding the use of these corpora, are included in this paper.One of the important roles and fundamental tasks of the corpus, which is the tokenization, is also presented in this paper.The development of the Malay language tokenizer is based on the syntactic data compatibility of Malay words written in Jawi.A series of experiments were performed to validate the corpus and to fulfill the requirement of the Jawi script tokenizer with an average error rate of 0.020255.Based on this promising result, the token will be used for the disambiguation and unknown word resolution, such as out-of vocabulary (OOV) problem in the tagging process.
Highlights
The annotated corpora, the Malay corpus (Knowles & Don, 2003) and the Dewan Bahasa dan Pustaka (DBP) database (DBP, 2015) corpora, are among the most valuable resources in current natural language processing in Malay studies
To evaluate the new corpus, a pilot test was conducted on word tokenization, which is the first step of any kind of natural language text preparation
A corpora that contains three Malay subcorpora is described in this paper. This corpora is unique due to the code applied. This corpora will serve as a benchmarking corpus for the development and evaluation systems in word tokenization, as well as further language processing
Summary
The annotated corpora, the Malay corpus (Knowles & Don, 2003) and the Dewan Bahasa dan Pustaka (DBP) database (DBP, 2015) corpora, are among the most valuable resources in current natural language processing in Malay studies. They underlie statistical research in monolingual tasks, such as sentence structure, syntactic disambiguation, semantic recognition, information retrieval, etc. For the Malaysian Malay community, the creation of a monolingual parallel corpus between Roman and Jawi has a special significance It provides the basis for the development of Malay language applications that can be used to facilitate or even avoid labor and time-consuming processes of manual handling of parallel language information. To evaluate the new corpus, a pilot test was conducted on word tokenization, which is the first step of any kind of natural language text preparation
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
