Sign Language Translation System Research Articles

Automated Arabic-Arabic sign language translation system based on 3D avatar technology

<p><span>Arabic sign language (ArSL) is the natural language of the deaf community in Arabic countries. Deaf people have a set of difficulties due to poor services available. They have problems accessing essential information or receiving an education, communicating with other communities, and engaging in activities. Thus, a machine translation system of Arabic to ArSL has been developed using avatar technologies. Firstly, a dictionary of ArSL was constructed using eSign editor Software. The constructed dictionary has three thousand signs. It can be adopted for the translation system in which written text can be transformed into sign language. The dictionary will be available as a free resource for researchers. It is complex and time-consuming, but it is an essential step in the machine translation of whole Arabic text to ArSL with 3D animations. Secondly, the translator has been developed. It performs syntactic and morphological analysis and then applies a set of rules to translate an Arabic text into ArSL text based on the structure and grammar of ArSL. The system is evaluated according to the parallel corpus that consists of 180 sentences using the metric for evaluation of translation with explicit ordering </span><span lang="IN">metric for evaluation of translation with explicit ordering</span><span>(METEOR) our system achieves a relative score of (86%).</span></p>

Sign Language Translation Systems

Sign language, often termed “dactylology,” is a mode of communication for those who are hard of hearing. With over 2.5 billion people projected to have hearing loss by 2050, there are very few efficient real-time sign language translation (SLT) applications present today despite extensive research in the domain. The main purpose of the systematic literature review is to analyze existing research in SLT systems and obtain results that will help in building an efficient and improved SLT system. A total of 125 different research articles within the time frame of 2015–2022 were identified. The study analyzes each paper against nine main research questions. The results obtained show the unique strengths and weaknesses of the different methods used, and while the reviewed papers showed significant results, there is still room for improvement in the implementations. This systematic literature review helps in identifying suitable methods to develop an efficient SLT application, identifies research gaps in this domain, and simultaneously indicates recent trends in the field of SLT systems.

Russian Sign Language Database for Clinical Use: Data and Annotation Peculiarities

This work has been carried out by the members of the Laboratory of Speech and Multimodal Interfaces of the St. Petersburg Federal Research Center of the Russian Academy of Sciences within an interdisciplinary research project aimed at the creation of an automatic Russian sign language translation system. The paper presents the design of a Russian sign language digital database for a specific subject area, namely, «The first visit to doctor».Our database is meant to be used first of all a dataset for training neural network-based systems of automatic translation from Russian sign language. But also, it can be of interest for linguistics of sign languages in general since the approach elucidates solution of a number of distorting phenomena typical for continuous sign language tracking, such as epenthesis, assimilation, reduction, and hold deletion.The principal difference between the presented video data and other datasets developed for similar purposes is the use of continuous sign utterances and elements of Russian sign language proper instead of the so-called “signed Russian” (that is, the visual form of the Russian spoken language), popular in deaf schooling.One of the most challenging problems, dealt with in the paper, is an automatic segmentation of sign speech into separate meaningful units with clearly defined boundaries. Its solution is hampered by signs’ deformations in continuous speech. In this paper, the segmentation is carried out within the framework of the movement-hold model. This approach allows extraction of their functional core as well as annotation of possible changes likely to appear in different signs’ segments. Accordingly, each utterance was subdivided into segments of motion and hold, and the resulting sign schemes were then entered into a separate directory containing information about the main parameters of each hold, including the possibilities of change due to immediate environment. The result of this work is a full decomposition of the signs, forming the database which can find its application in different statistical models of Russian Sign Language.Among other linguistic peculiarities of the database should be noted lexical variability of signs, mouthing and code switching between Russian sign language and “signed Russian”. Also, our signers learned Russian sign language in different regions of Russia, thus the collected data is potentially a source for research in dialectal variability of Russian sign language.

Translating Speech to Indian Sign Language Using Natural Language Processing

Language plays a vital role in the communication of ideas, thoughts, and information to others. Hearing-impaired people also understand our thoughts using a language known as sign language. Every country has a different sign language which is based on their native language. In our research paper, our major focus is on Indian Sign Language, which is mostly used by hearing- and speaking-impaired communities in India. While communicating our thoughts and views with others, one of the most essential factors is listening. What if the other party is not able to hear or grasp what you are talking about? This situation is faced by nearly every hearing-impaired person in our society. This led to the idea of introducing an audio to Indian Sign Language translation system which can erase this gap in communication between hearing-impaired people and society. The system accepts audio and text as input and matches it with the videos present in the database created by the authors. If matched, it shows corresponding sign movements based on the grammar rules of Indian Sign Language as output; if not, it then goes through the processes of tokenization and lemmatization. The heart of the system is natural language processing which equips the system with tokenization, parsing, lemmatization, and part-of-speech tagging.

A Survey on Machine Learning Based Techniques for Sign Language Translation System

Abstract: Sign language is the communication method used to bridge the gap among mute and deaf people and the rest of the people. But most people are not familiar with sign language, so communication becomes difficult among mute, deaf and the rest of the people. Improvement in machine learning technology has led to research on its application in sign language recognition and translation system. This paper presents a review of different sign language recognition and translation systems and the various techniques like classification algorithms that were implemented in them. We have found that although many sign language recognition and translation system exist none of them provides efficient and real-time conversation of sign language into audio and many sign language recognition and translation system are hand movement sensor-based which are not affordable to everyone. So, we have also discussed a sign language translation system integrated into a video calling application, in which we will be using a single shot multibox detector for hand detection, for feature extraction Inception V3 will be used, and support vector machines for classification, attention network for generating semantically correct sentences and a text to speech synthesizer for audio output. This system will be easy to access and affordable as it use

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions.

A sign language translation system can break the communication barrier between hearing-impaired people and others. In this paper, a novel American sign language (ASL) translation method based on wearable sensors was proposed. We leveraged inertial sensors to capture signs and surface electromyography (EMG) sensors to detect facial expressions. We applied a convolutional neural network (CNN) to extract features from input signals. Then, long short-term memory (LSTM) and transformer models were exploited to achieve end-to-end translation from input signals to text sentences. We evaluated two models on 40 ASL sentences strictly following the rules of grammar. Word error rate (WER) and sentence error rate (SER) are utilized as the evaluation standard. The LSTM model can translate sentences in the testing dataset with a 7.74% WER and 9.17% SER. The transformer model performs much better by achieving a 4.22% WER and 4.72% SER. The encouraging results indicate that both models are suitable for sign language translation with high accuracy. With complete motion capture sensors and facial expression recognition methods, the sign language translation system has the potential to recognize more sentences.

Wearable sign language translation system using strain sensors

A wearable-type sign language translation system using a strain sensor was presented. The system can recognize and display the sign language representing all 26 English alphabets and is devised for interpreting hand gesture of finger positions. The desired sensor performance is achieved by using a composite material that is formed by adding carbon nanoparticles to the Dragon skin as an elastomer. The strain sensor has stable piezoresistive characteristics and possesses sufficient characteristics to distinguish the differences between hand signs for full expressions of English alphabets without degrading the stretchability of the elastomer substrate. The wearable system is demonstrated with an assembly of the strain sensors, which achieves interpretation of the letters from the hand signs in real time.

Estimation of Hidden Neuron Requisite for Predictive Conversion from Sign Language to a Formal Language

Using sign language for communication has existed since millennia and has evolved over the ages to support the hearing impaired to a great extent. There have been numerous wonderful inventions and innovations aimed at improving the living conidtions of the hearing or speech impaired using technology which was contemporary. However, till date there exists no standard that can be shown as a single universal sign language translation system. This is partially due to the research happening in small pockets and lack of technology penetration in some countries. Largely the efforts to design translation are concentrated on Americal Sign Language (ASL) and Thai Sign Language (TSL). Other than the two, Indonasian Sign Language (SIBI) is the front runner in standardising a translation system. In our previous paper we had attempted to develop an ANN based translator for ASL. The system performed as per expectations. However, since the number of hidden neurons was taken as a variable, the estimation still remained a challenge.In this paper, we have developed a system to estimate the number of hidden neurons required for the accurate translation of ASL

Создание анимированного персонажа для системы компьютерного сурдоперевода

In recent years, much attention has been paid to people with disabilities around the world. Creating a barrier-free environment and the necessary conditions for their training and communication is one of the important tasks of any modern state. Since everyone should have equal opportunities to communicate with the outside world. In our country, it is also important and necessary to create a system of automatic sign language translation from Kazakh to sign language for people with hearing disabilities. This article reveals the prospects of using an animated character (avatar) to demonstrate gestures, the features of creating an animated character and applying sign language notation to control it. The avatar is aimed at showing the results of translating the Kazakh text into sign language based on the previously created dictionary of notation of gestures of the Kazakh sign language (KSL). When developing an automatic sign language translation system, the meaning of the translated text is revealed, it is translated into the grammar of the KSL. The compiled dictionary is then used to search for words in the dictionary, present them as notation, and display them using an avatar. To create an avatar model, software tools such as MakeHuman, Photoshop, and Blender are used in stages. Gesture animations are set up in the modern cross-platform graphics package Unity3D using libraries of "clips" created based on the dictionary of notations of gestures of the KSL.

Sign Language Recognition System Using Cnn

In this study, we have built an automatic sign language translation system for deaf and dumb persons to communicate with ordinary people. According to the Statistics Department of the Taiwan Ministry of Health and Welfare, there are 119,682 hearing impaired persons, and 14,831 voice function or language dysfunctions. The Deaf and dumb persons’ account for 11.7% of the population with physical and mental disabilities. However, there are only 488 qualified people with the sign language translation skill certificate, which shows the importance of automatic sign language translation systems. This system collects 11 signals including five fingers’ curvature, 3-axis gyroscope and 3-axis accelerometer from left and right hand separately. In addition, a total of 22 signals are collected by the two sensors, Flex sensor and GY-521 six-axis with single-board computer Arduino MEGA 2560; and then uploaded to server via ESP-01S Wi-Fi module. While server receives the 22 signals, it converts to a RGB picture using PHP program. As a result, we can compare the picture with the model trained by TensorFlow and the compared result is stored in the database. Meanwhile, the comparison stored in database which can be accessed by APP programs would be displayed on the screen of the mobile device and be read aloud. The TensorFlow training model collects 25 sign language gestures, each based on 100 training gesture pictures, and a sign language recognition training model is Convolutional Neural Network (CNN). In this study, the results of the sign language recognition training model are further confirmed by 10 people other than those in training database. So far, the indeed recognition rate of sign language is about 84.4%, and the system response time is about 2.243 seconds.

Recognition and Classification of Dynamic Hand Gestures by a Wearable Data-Glove

This paper introduces a complete work-flow for the translation of dynamic isolated signs based on data acquired from a data-glove. A sign language translation system based on a wearable device represents indeed a more efficient solution with respect to cameras or position trackers for helping speech-impaired people on a daily basis. The paper describes the different steps required for a sign language translation, namely segmentation, feature extraction and classification, together with the custom data-glove used for data-acquisition. The paper presents also experimental results, comparing different machine learning classifiers and discussing their performances both in terms of translation accuracy and computational time. The proposed work-flow has been tested both on data acquired from a custom data-glove and on a public database, and it outperforms those of other works in literature. The reported analysis suggests a multi-layer perceptron neural network as the most suitable classifier for the realization of a wearable sign language translation system.

İşitme Yetersizliği Yaşayanların İletişimlerine Yönelik Bir İşaret Dili Çeviri Sisteminin Geliştirilmesi ve Uzman Değerlendirmesi

Bu çalışmanın amacı, iletişim ortamı olarak geliştirilen prototip İDÇS (İşaret Dili Çeviri Sistemi)’nin iletişim sürecini ve akademik başarıyı destekleme potansiyelini belirlemektir. Çalışma nitel araştırma yöntemlerinden biri olan durum çalışması ile yürütülmüştür. Araştırmanın çalışma grubunu işitme engelliler öğretmenliğinden mezun özel eğitim öğretmenleri, uzman öğreticiler ve işaret dili tercümanları olmak üzere 11 uzman oluşturmaktadır. Geliştirilen sistemle ilgili uzman görüşleri alınmış ve toplanan veriler analiz edilmiştir. Elde edilen veriler iletişim süreçleri ve akademik başarı olmak üzere iki tema altında toplanmıştır. İletişim süreçlerine yönelik bulgular incelendiğinde katılımcılar tarafından İDÇS’nin en çok iletişim performansını ve sosyalleşmeyi arttırma potansiyelinin olduğu düşünülmektedir. Bununla birlikte İDÇS’nin başlangıç seviyesinde öğrenme, kelime dağarcığını artırma potansiyelinin olduğu da katılımcılar tarafından ifade edilmiştir. İDÇS’nin ders çalışma süreçlerine etki potansiyelinin bulunması da elde edilen diğer bulgular arasındadır. İDÇS’nin iletişim performansını, sosyalleşmeyi, kelime dağarcığını, akademik başarıyı, başlangıç seviyesinde işaret dili öğrenmeyi ve işaret dili öğrenmeye ilgiyi arttırma potansiyelinin olduğuna ilişkin görüşler bildirilmiştir. İşitme yetisinde oluşturabileceği tembelliğin soyutlanmaya yol açma ihtimalinin bulunduğu da elde edilen sonuçlar arasındadır.

Development of a New Sign Language Translation System for People with Autism Spectrum Disorder

Many people diagnosed with autism spectrum disorder (ASD) have difficulty in developing communication skills, and they suffer from speech disorders. They must use an augmentative and alternative communication (AAC) system that helps with ordinary speech, such as picture exchange communication systems, text systems, and voice output devices. Among them, sign language has been recommended for people with ASD because it has many advantages as a communication system. This paper describes the possible impact of a new sign language translation device, called Talking Hands, for people with ASD. This paper describes Talking Hands from a technological point of view, focusing on its features that can help people ASDs with speech impairment. Taking into account the most recent studies on sign language application on people with disabilities, we analyse the advantages that Talking Hands could give to people with speech impairment, increasing the ones naturally given by sign languages. Although there are still some limitations in the use of Talking Hands on a daily basis, the analysis indicates Talking Hands is an innovative AAC device for people with ASD and could have a positive impact on others with speech disorders.

Automated Bangla sign language translation system for alphabets by means of MobileNet

Individuals with hearing and speaking impairment communicate using sign language. The movement of hand, body and expressions of face are the means by which the people, who are unable to hear and speak, can communicate. Bangla sign alphabets are formed with one or two hand movements. There are some features which differentiates the signs. To detect and recognize the signs, analyzing its shape and comparing its features is necessary. This paper aims to propose a model and build a computer systemthat can recognize Bangla Sign Lanugage alphabets and translate them to corresponding Bangla letters by means of deep convolutional neural network (CNN). CNN has been introduced in this model in form of a pre-trained model called “MobileNet” which produced an average accuracy of 95.71% in recognizing 36 Bangla Sign Language alphabets.

Understanding vision-based continuous sign language recognition

Real-time sign language translation systems, that convert continuous sign sequences to text/speech, will facilitate communication between the deaf-mute community and the normal hearing majority. A translation system could be vision-based or sensor-based, depending on the type of input it receives. To date, most of the commercial systems for this purpose are sensor-based, which are expensive and not user-friendly. Vision-based sign translation systems are the need of the hour but should overcome many challenges to build a full-fledged working system. Preliminary investigations in this work have revealed that the traditional approaches to continuous sign language recognition (CSLR) using HMM, CRF and DTW, tried to solve the problem of Isolated Sign Language Recognition (ISLR) and extended the solution to CSLR, leading to reduced performance. The main challenge of identifying Movement Epenthesis (ME) segments in continuous utterances, were handled explicitly with these traditional methods. With the advent of technologies like Deep Learning, more feasible solutions for vision-based CSLR are emerging, which has led to an increase in the research on vision-based approaches. In this paper, a detailed review of all the works in vision-based CSLR is presented, based on the methods they have followed. The challenges posed in continuous sign recognition are also discussed in detail, followed by a brief on sensor-based systems and benchmark databases. Finally, performance evaluation of all the associated methods are performed, which leads to a short discussion on the overall study and concludes by pointing out future research directions in the field.

Sign Language Translation Using Deep Convolutional Neural Networks

Sign language is a natural, visually oriented and non-verbal communication channel between people that facilitates communication through facial/bodily expressions, postures and a set of gestures. It is basically used for communication with people who are deaf or hard of hearing. In order to understand such communication quickly and accurately, the design of a successful sign language translation system is considered in this paper. The proposed system includes object detection and classification stages. Firstly, Single Shot Multi Box Detection (SSD) architecture is utilized for hand detection, then a deep learning structure based on the Inception v3 plus Support Vector Machine (SVM) that combines feature extraction and classification stages is proposed to constructively translate the detected hand gestures. A sign language fingerspelling dataset is used for the design of the proposed model. The obtained results and comparative analysis demonstrate the efficiency of using the proposed hybrid structure in sign language translation.

СЛОВАРИ КАЗАХСКОГО ЯЗЫКА КАК ОСНОВА СЕМАНТИЧЕСКОГО АНАЛИЗА В СИСТЕМЕ КОМПЬЮТЕРНОГО СУРДОПЕРЕВОДА

his article discusses the development of a semantic dictionary of the Kazakh language for a computer translation system from Kazakh to Kazakh sign language, which will take into account the semantics of the Kazakh language and the Kazakh sign language. The semantic dictionary of the Kazakh language serves as the basis of computer translation technology from the Kazakh language to the Kazakh sign language. In the future, it will allow semantic analysis of the source text. The authors of the article analyzed and selected the available dictionaries of the Kazakh language used in the development of the semantic dictionary database. Dictionaries of the Kazakh language provide an opportunity for computer-based sign language translation of the Kazakh sign language. The article also presents the possibility of using L. S. Dimskis notation to develop a dictionary of the structure of gestures of the Kazakh sign language. The prospect of its inclusion in the database of semantic dictionary is revealed. And also revealed the need for a dictionary of gestures in the development of automated sign language translation system as a whole, taking into account its effectiveness and the possibility of full practical use.

Sign Language Semantic Translation System using Ontology and Deep Learning

Translation and understanding sign language may be difficult for some. Therefore, this paper proposes a solution to this problem by providing an Arabic sign language translation system using ontology and deep learning techniques. That is to interpret user’s signs to different meanings. This paper implemented ontology on the sign language domain to solve some sign language challenges. In this first version, simple static signs composed of Arabic alphabets and some Arabic words started to translate. Deep Convolution Neural Network (CNN) architecture was trained and tested on a pre-made Arabic sign language dataset and on a dataset collected in this paper to obtain better accuracy in recognition. Experimental results show that according to the pre-made Arabic sign language dataset the classification accuracy of the training set (80% of the dataset) was 98.06% and recognition accuracy of the testing set (20% of the dataset) was 88.87%. According to the collected dataset, the classification accuracy of the training set was 98.6% and Semantic recognition accuracy of the testing set was 94.31%.

Anlam Çözümleme Modelleri İle Geliştirilen Türkçe İşaret Dili Çeviri Sisteminin Nesnelerin İnterneti Üzerinden Web Ortamında Uygulaması

Günlük hayatta insanlar; duygularını, düşüncelerini ve yaşantılarını dili kullanarak ya da kullanmaksızın çevrelerindeki insanlara aktararak iletişime geçerler. Dilsel ve işitsel yetisi olmayan kişiler ise çevreleriyle bu iletişimi işaret dilini kullanarak gerçekleştirirler. İşaret dili ise, engelli kişilerin hem kendi aralarında hem de bu dili bilen engelsiz kişilerle jest, mimik ve vücut hareketleriyle anlaşmalarını sağlayan bir dil olarak tanımlanır. Bu anlamda Nesnelerin İnterneti üzerinden anlam çözümlenme modelleri ile web (ağ) ortamında çalışmasına yönelik geliştirilen bu çalışma, duyma ve konuşma bozukluğu olan kişilerle işaret dilini bilmeyenler arasındaki iletişimi sağlayan bir Türkçe İşaret Dili Çeviri Sistemidir. Gerçekleştirilen bu sistemle; işaret dilini gösteren el hareketleri, derinlik kamerasıyla yakalanmış, MQTT (Message Queuing Telemetry Transport /Sıralı Telemetri Mesaj İletimi) sunucusuna gönderilmiştir. Gönderilen bu görüntüler Üç Boyutlu Modelleme ve Hareket Analizinde kullanılan yapay sinir ağları algoritmaları ve makine öğrenmesi yöntemleri ile analiz edilmiştir. Analiz edilen görüntülerden elde edilen sözcüklerin gerçek anlamları ise, Biçimsel Kavram Analizi Teorisi ile hazırlanan tematik rollerden oluşan gerçeklik modelleriyle bulunmuştur. Gerçek anlamları bulunan bu kelimeler, içinde geçtiği cümle ile birlikte lokal ağ ortamında yayınlanmıştır. Böylece, engelli bir kişi aynı veya farklı mekanlarda yer alan kişi veya kişilerle iletişim kurabilmiştir.

Neural Sign Language Translation Based on Human Keypoint Estimation

We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (Korea Electronics Technology Institute) sign language dataset, which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting point for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from the face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieved 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compared several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.