Sign Language Machine Translation Research Articles

From rule-based models to deep learning transformers architectures for natural language processing and sign language translation systems: survey, taxonomy and performance evaluation

With the growing Deaf and Hard of Hearing population worldwide and the persistent shortage of certified sign language interpreters, there is a pressing need for an efficient, signs-driven, integrated end-to-end translation system, from sign to gloss to text and vice-versa. There has been a wealth of research on machine translations and related reviews. However, there are few works on sign language machine translation considering the particularity of the language being continuous and dynamic. This paper aims to address this void, providing a retrospective analysis of the temporal evolution of sign language machine translation algorithms and a taxonomy of the Transformers architectures, the most used approach in language translation. We also present the requirements of a real-time Quality-of-Service sign language machine translation system underpinned by accurate deep learning algorithms. We propose future research directions for sign language translation systems.

A deaf-centred art-science approach to community engagement with sign language technologies

Sign Language Machine Translation (SLMT) is an emerging technology that will primarily impact deaf and hard of hearing communities. Historically, these technologies have been developed by hearing scientists, without input from deaf researchers or community representatives. SignON was a three-year project exploring the application of SLMT, and using a co-creation approach which was led by the European Union of the Deaf. Here we describe an art-science method to engage deaf, hard of hearing, and hearing audiences with SLMT through theatre and performance. We also reflect on the insights, perspectives, and feedback shared by audience members.

Marathi to Indian Sign Language Machine Translation

Machine translation has been a prominent field of research, contributing significantly to human life enhancement. Sign language machine translation, a subfield, focuses on translating spoken language content into sign language and vice versa, thereby facilitating communication between the normal hearing and hard-of-hearing communities, promoting inclusivity. This study presents the development of a ‘sign language machine translation system’ converting simple Marathi sentences into Indian Sign Language (ISL) glosses and animation. Given the low-resource nature of both languages, a phrase-level rule-based approach was employed for the translation. Initial encoding of translation rules relied on basic linguistic knowledge of Marathi and ISL, with subsequent incorporation of rules to address 'simultaneous morphological' features in ISL. These rules were applied during the ‘generation phase’ of translation to dynamically adjust phonological sign parameters, resulting in improved target sentence fluency. The paper provides a detailed description of the system architecture, translation rules, and comprehensive experimentation. Rigorous evaluation efforts were undertaken, encompassing various linguistic features, and the findings are discussed herein. The web-based version of the system serves as an interpreter for brief communications and can support the teaching and learning of sign language and its grammar in schools for hard-of-hearing students.

A Novel Natural Language Processing (NLP)–Based Machine Translation Model for English to Pakistan Sign Language Translation

The deaf community in the world uses a gesture-based language, generally known as sign language. Every country has a different sign language; for instance, USA has American Sign Language (ASL) and UK has British Sign Language (BSL). The deaf community in Pakistan uses Pakistan Sign Language (PSL), which like other natural languages, has a vocabulary, sentence structure, and word order. Majority of the hearing community is not aware of PSL due to which there exists a huge communication gap between the two groups. Similarly, deaf persons are unable to read text written in English and Urdu. Hence, the provision of an effective translation model can support the cognitive capability of the deaf community to interpret natural language materials available on the Internet and in other useful resources. This research involves exploiting natural language processing (NLP) techniques to support the deaf community by proposing a novel machine translation model that translates English sentences into equivalent Pakistan Sign Language (PSL). Though a large number of machine translation systems have been successfully implemented for natural to natural language translations, natural to sign language machine translation is a relatively new area of research. State-of-the-art works in natural to sign language translation are mostly domain specific and suffer from low accuracy scores. Major reasons are specialised language structures for sign languages, and lack of annotated corpora to facilitate development of more generalisable machine translation systems. To this end, a grammar-based machine translation model is proposed to translate sentences written in English language into equivalent PSL sentences. To the best of our knowledge, this is a first effort to translate any natural language to PSL using core NLP techniques. The proposed approach involves a structured process to investigate the linguistic structure of PSL and formulate the grammatical structure of PSL sentences. These rules are then formalised into a context-free grammar, which, in turn, can be efficiently implemented as a parsing module for translation and validation of target PSL sentences. The whole concept is implemented as a software system, comprising the NLP pipeline and an external service to render the avatar-based video of translated words, in order to compensate the cognitive hearing deficit of deaf people. The accuracy of the proposed translation model has been evaluated manually and automatically. Quantitative results reveal a very promising Bilingual Evaluation Understudy (BLEU) score of 0.78. Subjective evaluations demonstrate that the system can compensate for the cognitive hearing deficit of end users through the system output expressed as a readily interpretable avatar. Comparative analysis shows that our proposed system works well for simple sentences but struggles to translate compound and compound complex sentences correctly, which warrants future ongoing research.

Support of Arabic Sign Language Machine Translation based on Morphological processing

Support of Arabic Sign Language Machine Translation based on Morphological processing

English text to Indian Sign Language Machine Translation: A Rule Based Method

The hearing challenged community all over world face difficulties to communicate with others. Machine translation has been one of the prominent technologies to facilitate a two way communication to the deaf and hard of hearing community all over the world. We have explored and formulated the fundamental rules of Indian Sign Language and implemented as a translation mechanism of English Text to Indian sign Language glosses. The structure of the source text is identified and transferred to the target language according to the formulated rules and sub rules. The intermediate phases of the transfer process is also mentioned in this research work.

Early estimation model for 3D-discrete indian sign language recognition using graph matching

Machine translation of sign language is a critical task of computer vision. In this work, we propose to use 3D motion capture technology for sign capture and graph matching for sign recognition. Two problems related to 3D sign matching are addressed in this work: (1) how to identify same signs with different number of motion frames and (2) sign extraction from a clutter of non-sign hand motions. These two problems make the 2D or 3D sign language machine translation a challenging task. We propose graph matching with early estimation model to address these problems in two phases. The first phase consists of intra graph matching for motion frame extraction, which retains motion intensive frames in database and query 3D videos. The second phase applies inter graph matching with early estimation model on motion extracted query and dataset 3D videos. The proposed model increases the speed of the graph matching algorithm in estimating a sign with fewer frames. To test the graph matching model, we recorded 350 words of Indian sign language with 3D motion capture technology. For testing 4 variations per sign are captured for all signs with 5 different signers at same, slower, faster hand speeds and sign mixed cluttered hand motions. The early estimation graph matching model is tested for accuracy and efficiency in classifying 3D signs with the two induced real time constraints. In addition to 3D sign language dataset, the proposed method is validated on five benchmark datasets and against the state-of-the-art graph matching methods.

ODIA LANGUAGE TO SIGN LANGUAGE MACHINE TRANSLATION SYSTEM: A FACE-TUBE FOR AUDIBLE IMPAIRED PEOPLE.

25Apr 2017 ODIA LANGUAGE TO SIGN LANGUAGE MACHINE TRANSLATION SYSTEM: A FACE-TUBE FOR AUDIBLE IMPAIRED PEOPLE. Rudranarayan Mohapatra and Anadi Charan Gan. P.G. Department of Odia, Utkal University, Vanivihar, Bhubaneswar ? 71004

Learning sign language machine translation based on elastic net regularization and latent semantic analysis

In this paper, we present a new sign language machine translation approach based on regression method. The aim of this work is to improve the translation quality and accuracy of existing regularized regression methods. Our approach represents a methodological foundation for small-scale corpus domains such as the Sign Language Machine Translation field. Our method is based on the Elastic net regularization using linear combination of the L1 and L2 penalties of the lasso and ridge methods. We show that using both the de-bruijn graph with the Latent Semantic Analysis technique in the decoding process improves the translation results. The system is experimented on American Sign Language parallel corpora containing 300 sentences and assessed by BLEU, METEOR, NIST and F1-MESURE machine translation evaluation metrics. We obtained good experimental results compared to classical phrase based approach i.e MOSES framework. Also our approach improved the translation results compared to LASSO and RIDGE regression approaches.

Analysis, preparation, and optimization of statistical sign language machine translation

Sign languages represent an interesting niche for statistical machine translation that is typically hampered by the scarceness of suitable data, and most papers in this area apply only a few, well-known techniques that are not adapted to small-sized corpora. In this article, we analyze existing data collections and emphasize their quality and usability for statistical machine translation. We also offer findings in the proper preprocessing of a sign language corpus, by introducing sentence end markers, splitting compound words and handling parallel communication channels. Then, we focus on optimization procedures that are tailored to scarce resources, such as scaling factor optimization, alignment optimization and system combination. All methods are evaluated on two of the largest sign language corpora available.

Проблема автоматизации сурдоперевода с позиции прикладной лингвистики

Проблема автоматизации сурдоперевода с позиции прикладной лингвистики

Automatic categorization for improving Spanish into Spanish Sign Language machine translation

This paper describes a preprocessing module for improving the performance of a Spanish into Spanish Sign Language (Lengua de Signos Española: LSE) translation system when dealing with sparse training data. This preprocessing module replaces Spanish words with associated tags. The list with Spanish words (vocabulary) and associated tags used by this module is computed automatically considering those signs that show the highest probability of being the translation of every Spanish word. This automatic tag extraction has been compared to a manual strategy achieving almost the same improvement. In this analysis, several alternatives for dealing with non-relevant words have been studied. Non-relevant words are Spanish words not assigned to any sign. The preprocessing module has been incorporated into two well-known statistical translation architectures: a phrase-based system and a Statistical Finite State Transducer (SFST). This system has been developed for a specific application domain: the renewal of Identity Documents and Driver's License. In order to evaluate the system a parallel corpus made up of 4080 Spanish sentences and their LSE translation has been used. The evaluation results revealed a significant performance improvement when including this preprocessing module. In the phrase-based system, the proposed module has given rise to an increase in BLEU (Bilingual Evaluation Understudy) from 73.8% to 81.0% and an increase in the human evaluation score from 0.64 to 0.83. In the case of SFST, BLEU increased from 70.6% to 78.4% and the human evaluation score from 0.65 to 0.82.

A prototype Greek text to Greek Sign Language conversion system

In this article, a prototype Greek text to Greek Sign Language (GSL) conversion system is presented. The system is integrated into an educational platform that addresses the needs of teaching GSL grammar and was developed within the SYNENNOESE project (Efthimiou et al. 2004a. Developing an e-learning platform for the Greek sign language. In: K. Miesenberger, J. Klaus, and W. Zagler, eds. Computer helping people with special needs, Lecture Notes in Computer Science, Springer-Verlag, vol. 3118, 1107–1113). The detailed implementation of the language-processing component of a Greek text to GSL conversion system is provided, focusing upon the inherent problems of knowledge elicitation of sign language (SL) grammar and its implementation within a parser framework. It is based on an SL dictionary (Efthimiou et al. 2004a) database of coded GSL knowledge. The proposed system has been designed and implemented after considering most state-of-the-art SL machine translation or Conversion systems, such as Vsigns (Papadogiorgaki et al. 2004. VSigns – a virtual sign synthesis web tool. In: Proceedings of Sixth COST 276 Workshop on Information and Knowledge Management for Integrated Media Communication, May 2004, Thessaloniki, Greece), ZARDOZ (Veale et al. 1998. The challenges of cross-modal translation: English to sign language translation in the ZARDOZ system. Machine Translation, 13, 81–106) and SignSynth (Angus 2001. SignSynth: a sign language synthesis application using Web3D and Perl. In: Gesture and Sign Lanauage in Human–Computer Interaction. London, UK: International Gesture Workshop), and taking into account their advantages and disadvantages. The overall architecture is innovative since other existing systems either do not consider the GSL or they cannot be effectively applied on sentences but just on single words. The system is demonstrable on any conventional PC.

American Sign Language natural language generation and machine translation

Although deaf students in the U.S. and Canada are taught written English, their inability to hear spoken English results in most deaf U.S. high school graduates (18 year olds) reading at a fourth-grade (10 year old) level (Holt, 1991). Unfortunately, many deaf accessibility aids, like television closed captioning or teletype telephones, assume the user has strong English literacy skills. Many deaf people with English reading difficulty are fluent in American Sign Language (ASL); so, an English-to-ASL automated machine translation (MT) system could make information and services accessible when English text captioning is too complex or an interpreter is unavailable.

Process of research and development on human interface. Development of sign language machine translation.

Process of research and development on human interface. Development of sign language machine translation.

Study on translating Chinese into Chinese sign language

Sign language is a visual-gestural language mainly used by hearing-impaired people to communicate with each other. Gesture and facial expression are important grammar parts of sign language. In this paper, a text-based transformation method of Chinese-Chinese sign language machine translation is proposed. Gesture and facial expression models are created. And a practical system is implemented. The input of the system is Chinese text. The output of the system is “graphics person” who can gesticulate Chinese sign language accompanied by facial expression that corresponds to the Chinese text entered so as to realize automatic translation from Chinese text to Chinese sign language.