A Novel Natural Language Processing (NLP)–Based Machine Translation Model for English to Pakistan Sign Language Translation

Abstract

The deaf community in the world uses a gesture-based language, generally known as sign language. Every country has a different sign language; for instance, USA has American Sign Language (ASL) and UK has British Sign Language (BSL). The deaf community in Pakistan uses Pakistan Sign Language (PSL), which like other natural languages, has a vocabulary, sentence structure, and word order. Majority of the hearing community is not aware of PSL due to which there exists a huge communication gap between the two groups. Similarly, deaf persons are unable to read text written in English and Urdu. Hence, the provision of an effective translation model can support the cognitive capability of the deaf community to interpret natural language materials available on the Internet and in other useful resources. This research involves exploiting natural language processing (NLP) techniques to support the deaf community by proposing a novel machine translation model that translates English sentences into equivalent Pakistan Sign Language (PSL). Though a large number of machine translation systems have been successfully implemented for natural to natural language translations, natural to sign language machine translation is a relatively new area of research. State-of-the-art works in natural to sign language translation are mostly domain specific and suffer from low accuracy scores. Major reasons are specialised language structures for sign languages, and lack of annotated corpora to facilitate development of more generalisable machine translation systems. To this end, a grammar-based machine translation model is proposed to translate sentences written in English language into equivalent PSL sentences. To the best of our knowledge, this is a first effort to translate any natural language to PSL using core NLP techniques. The proposed approach involves a structured process to investigate the linguistic structure of PSL and formulate the grammatical structure of PSL sentences. These rules are then formalised into a context-free grammar, which, in turn, can be efficiently implemented as a parsing module for translation and validation of target PSL sentences. The whole concept is implemented as a software system, comprising the NLP pipeline and an external service to render the avatar-based video of translated words, in order to compensate the cognitive hearing deficit of deaf people. The accuracy of the proposed translation model has been evaluated manually and automatically. Quantitative results reveal a very promising Bilingual Evaluation Understudy (BLEU) score of 0.78. Subjective evaluations demonstrate that the system can compensate for the cognitive hearing deficit of end users through the system output expressed as a readily interpretable avatar. Comparative analysis shows that our proposed system works well for simple sentences but struggles to translate compound and compound complex sentences correctly, which warrants future ongoing research.