Sign Language Translation System Research Articles

From rule-based models to deep learning transformers architectures for natural language processing and sign language translation systems: survey, taxonomy and performance evaluation

With the growing Deaf and Hard of Hearing population worldwide and the persistent shortage of certified sign language interpreters, there is a pressing need for an efficient, signs-driven, integrated end-to-end translation system, from sign to gloss to text and vice-versa. There has been a wealth of research on machine translations and related reviews. However, there are few works on sign language machine translation considering the particularity of the language being continuous and dynamic. This paper aims to address this void, providing a retrospective analysis of the temporal evolution of sign language machine translation algorithms and a taxonomy of the Transformers architectures, the most used approach in language translation. We also present the requirements of a real-time Quality-of-Service sign language machine translation system underpinned by accurate deep learning algorithms. We propose future research directions for sign language translation systems.

Translating Gestures: Utilizing CNN to Enhance ASL Communication and Understanding

This research introduces a system for translating American Sign Language (ASL) utilising Convolutional Neural Networks. (CNNs). The system under consideration has the capability to identify manual gestures executed by persons communicating in American Sign Language (ASL) and subsequently convert them into written language, thereby facilitating uninterrupted communication between the deaf and hearing populations. The CNN model was validated and trained using a set of data made up of 78,300 pictures of movements in the ASL Alphabet written in American Sign Language (ASL). In order to improve the performance of the model, the pre-processing of the data included a number of stages, such as converting the photos to grayscale, normalising the pixel values, and enhancing performance of the model. In order to ease classification, fully connected layers were added after a succession of pooling and convolutional layers in CNN's design. After 15 epochs of training, the model attained a validation accuracy of 99.85%. The findings of this research demonstrate the viability of employing Convolutional Neural Networks (CNNs) in the creation of precise and effective American Sign Language (ASL) translation systems, which can serve as a means of communication for people with auditory impairments.

Sign-to-Text Translation from Panamanian Sign Language to Spanish in Continuous Capture Mode with Deep Neural Networks

Convolutional neural networks (CNN) have provided great advances for the task of sign language recognition (SLR). However, recurrent neural networks (RNN) in the form of long–short-term memory (LSTM) have become a means for providing solutions to problems involving sequential data. This research proposes the development of a sign language translation system that converts Panamanian Sign Language (PSL) signs into text in Spanish using an LSTM model that, among many things, makes it possible to work with non-static signs (as sequential data). The deep learning model presented focuses on action detection, in this case, the execution of the signs. This involves processing in a precise manner the frames in which a sign language gesture is made. The proposal is a holistic solution that considers, in addition to the seeking of the hands of the speaker, the face and pose determinants. These were added due to the fact that when communicating through sign languages, other visual characteristics matter beyond hand gestures. For the training of this system, a data set of 330 videos (of 30 frames each) for five possible classes (different signs considered) was created. The model was tested having an accuracy of 98.8%, making this a valuable base system for effective communication between PSL users and Spanish speakers. In conclusion, this work provides an improvement of the state of the art for PSL–Spanish translation by using the possibilities of translatable signs via deep learning.

Towards a Bidirectional Mexican Sign Language–Spanish Translation System: A Deep Learning Approach

People with hearing disabilities often face communication barriers when interacting with hearing individuals. To address this issue, this paper proposes a bidirectional Sign Language Translation System that aims to bridge the communication gap. Deep learning models such as recurrent neural networks (RNN), bidirectional RNN (BRNN), LSTM, GRU, and Transformers are compared to find the most accurate model for sign language recognition and translation. Keypoint detection using MediaPipe is employed to track and understand sign language gestures. The system features a user-friendly graphical interface with modes for translating between Mexican Sign Language (MSL) and Spanish in both directions. Users can input signs or text and obtain corresponding translations. Performance evaluation demonstrates high accuracy, with the BRNN model achieving 98.8% accuracy. The research emphasizes the importance of hand features in sign language recognition. Future developments could focus on enhancing accessibility and expanding the system to support other sign languages. This Sign Language Translation System offers a promising solution to improve communication accessibility and foster inclusivity for individuals with hearing disabilities.

Low hysteresis, water retention, anti-freeze multifunctional hydrogel strain sensor for human-machine interfacing and real-time sign language translation.

Hydrogel strain sensors have received increasing attention due to their potential applications in human-machine interfaces and flexible electronics. However, they usually suffer from both mechanical and electrical hysteresis and poor water retention, which limit their practical applications. To address this challenge, a poly(acrylic acid-co-acrylamide) hydrogel crosslinked by silica nanoparticles is fabricated via photo polymerization and salting-out of hydrophilic ions for the strain sensor. The resulting hydrogel strain sensor possessed low electrical hysteresis (1.6%), low mechanical hysteresis (<7%), high cycle stability (>10 000 cycles), high durability, water retention and anti-freezing ability. Moreover, this strain sensor can be used as a wearable sensor for real-time control of robotic hands and hand gesture recognition. Finally, a sign language translation system has been demonstrated with the aid of machine learning, achieving recognition rates of over 98% for 15 different sign languages. This work offers a promising prospect for human-machine interfaces, smart wearable devices, and the Internet of Things.

Bi-directional translation methods of ASL and speech/text

This paper introduces a sign language translation system designed to bridge the communication gap between Hard of Hearing (HOH) or mute individuals, and address challenges in job opportunities and mental health. The system offers bi-directional translation of sign language and speech/text. For sign language to speech/text translation, the system utilizes either camera-based gestures, or data-glove inputs. The reverse translation is facilitated through a 3D avatar demonstrating the corresponding sign language for the input speech or text. Our findings demonstrate the system’s translation accuracy and its potential to empower HOH and mute individuals in various communication scenarios.

Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method

Hand gesture recognition (HGR) is a primary mode of communication and human involvement. While HGR can be used to enhance user interaction in human-computer interaction (HCI), it can also be used to overcome language barriers. For example, HGR could be used to recognize sign language, which is a visual language expressed by hand movements, poses, and faces, and used as a basic communication mode by deaf people around the world. This research aims to create a new method to detect dynamic hand movements, poses, and faces in sign language translation systems. The Long Short-Term Memory Modification (LSTM) approach and the Mediapipe library are used to recognize dynamic hand movements. In this study, twenty dynamic movements that match the context were designed to solve the challenge of identifying dynamic signal movements. Sequences and image processing data are collected using MediaPipe Holistic, processed, and trained using the LSTM Modification method. This model is practiced using training and validation data and a test set to evaluate it. The training evaluation results using the confusion matrix achieved an average accuracy of twenty words trained, which was 99.4% with epoch 150. The results of experiments per word showed detection accurateness of 85%, while experiments using sentences only reached 80%. The research carried out is a significant step forward in advancing the accuracy and practice of the dynamic sign language recognition system, promising better communication and accessibility for deaf people.

Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method

Hand gesture recognition (HGR) is a primary mode of communication and human involvement. While HGR can be used to enhance user interaction in human-computer interaction (HCI), it can also be used to overcome language barriers. For example, HGR could be used to recognize sign language, which is a visual language expressed by hand movements, poses, and faces, and used as a basic communication mode by deaf people around the world. This research aims to create a new method to detect dynamic hand movements, poses, and faces in sign language translation systems. The Long Short-Term Memory Modification (LSTM) approach and the Mediapipe library are used to recognize dynamic hand movements. In this study, twenty dynamic movements that match the context were designed to solve the challenge of identifying dynamic signal movements. Sequences and image processing data are collected using MediaPipe Holistic, processed, and trained using the LSTM Modification method. This model is practiced using training and validation data and a test set to evaluate it. The training evaluation results using the confusion matrix achieved an average accuracy of twenty words trained, which was 99.4% with epoch 150. The results of experiments per word showed detection accurateness of 85%, while experiments using sentences only reached 80%. The research carried out is a significant step forward in advancing the accuracy and practice of the dynamic sign language recognition system, promising better communication and accessibility for deaf people.

Automatic sign language translation system using neural network technologies and 3D animation

Implementation of automatic sign language translation software in the process of social inclusion of people with hearing impairment is an important task. Social inclusion for people with hearing disabilities is an acute problem that must be solved in the context of the development of IT technologies and legislative initiatives that ensure the rights of people with disabilities and their equal opportunities. This substantiates the relevance of the research of assistive technologies, in the context of software tools, such as the process of social inclusion of people with severe hearing impairment in society. The subject of research is methods of automated sign language translation using intelligent technologies. The purpose of the work is the development and research of sign language automation methods to improve the quality of life of people with hearing impairments in accordance with the "Goals of Sustainable Development of Ukraine" (in the "Reduction of Inequality" part). The main tasks of the research are the development and testing of methods of converting sign language into text, converting text into sign language, as well as automating translation from one sign language to another sign language using modern intelligent technologies. Neural network modeling and 3D animation methods were used to solve these problems. The following results were obtained in the work: the main problems and tasks of social inclusion for people with hearing impairments were identified; a comparative analysis of modern methods and software platforms of automatic sign language translation was carried out; a system combining the SL-to-Text method is proposed and investigated; the Text-to-SL method using 3D animation to generate sign language concepts; the method of generating a 3D-animated gesture from video recordings; method of implementing the Sign Language1 to Sign Language2 technology. For gesture recognition, a convolutional neural network model is used, which is trained using imported and system-generated datasets of video gestures. The trained model has a high recognition accuracy (98.52%). The creation of a 3D model for displaying the gesture on the screen and its processing took place in the Unity 3D environment. The structure of the project, executive and auxiliary files used to build 3D animation for the generation of sign language concepts includes: event handler files; display results according to which they carry information about the position of the tracked points of the body; files that store the characteristics of materials that have been added to certain body mapping points. Conclusions: the proposed methods of automated translation have practical significance, which is confirmed by the demo versions of the software applications "Sign Language to Text" and "Text to Sign Language". A promising direction for continuing research on the topic of the work is the improvement of SL1-to-SL2 methods, the creation of open datasets of video gestures, the joining of scientists and developers to fill dictionaries with concepts of various sign languages.

Bioinspired Photoreceptors with Neural Network for Recognition and Classification of Sign Language Gesture.

This work addresses the design and implementation of a novel PhotoBiological Filter Classifier (PhBFC) to improve the accuracy of a static sign language translation system. The captured images are preprocessed by a contrast enhancement algorithm inspired by the capacity of retinal photoreceptor cells from mammals, which are responsible for capturing light and transforming it into electric signals that the brain can interpret as images. This sign translation system not only supports the effective communication between an agent and an operator but also between a community with hearing disabilities and other people. Additionally, this technology could be integrated into diverse devices and applications, further broadening its scope, and extending its benefits for the community in general. The bioinspired photoreceptor model is evaluated under different conditions. To validate the advantages of applying photoreceptors cells, 100 tests were conducted per letter to be recognized, on three different models (V1, V2, and V3), obtaining an average of 91.1% of accuracy on V3, compared to 63.4% obtained on V1, and an average of 55.5 Frames Per Second (FPS) in each letter classification iteration for V1, V2, and V3, demonstrating that the use of photoreceptor cells does not affect the processing time while also improving the accuracy. The great application potential of this system is underscored, as it can be employed, for example, in Deep Learning (DL) for pattern recognition or agent decision-making trained by reinforcement learning, etc.

Tamil Sign Language Translation and Recognition System for Deaf-mute People Using Image Processing Techniques

This work concentrates on the device, which helps as a translation system for translating sign gestures into text. The disabled people, in particular, hearing and speech impaired people, are facing difficulties in society. Communication of disabled people becomes worse as the majority of ordinary people do not understand it. These disabled people face difficulty communicating with others; some have many problems communicating with others in sign language. It causes the communication gap between them where impaired people cannot share their views and skills with others. We headed to facilitate communication between the disabled people and the "Tamil sign language translator to solve this problem." Here, gestures are translated to Tamil language to find a localized solution. It processes 31 Tamil alphabets, 12 Vowels, 18 Consonants, and 1 Aayudha Ezhuthu. It is 32 combinations with five fingers points either up or down and mapped to decimal numbers. Here in this process, we need edge detection, which is accurately done and Processed by canny edge detection. In addition to this process, we have used two gesture recognition methods and training the input system through our mainframe algorithm called scale-invariant feature detection transform. We have developed this system, which is useful for deaf and dumb people for essential communication.

A Survey of Advancements in Real-Time Sign Language Translators: Integration with IoT Technology

Real-time sign language translation systems are of paramount importance in enabling communication for deaf and hard-of-hearing individuals. This population relies on various communication methods, including sign languages and visual techniques, to interact with others. While assistive technologies, such as hearing aids and captioning, have improved their communication capabilities, a significant communication gap still exists between sign language users and non-users. In order to bridge this gap, numerous sign language translation systems have been developed, encompassing sign language recognition and gesture-based controls. Our research aimed to analyze the advancements in real-time sign language translators developed over the past five years and their integration with IoT technology. By closely examining these technologies, we aimed to attain a deeper comprehension of their practical applications and evolution in the domain of sign language translation. We analyzed the current literature, technical reports, and conference papers on real-time sign language translation systems. Our results offer insights into the current state of the art in real-time sign language translation systems and their integration with IoT technology. We also provide a deep understanding of the recent developments in sign language translation technology and the potential for their fusion with Internet of Things technology to improve communication and promote inclusivity for the deaf and hard-of-hearing population.

Ultra‐Robust and Sensitive Flexible Strain Sensor for Real‐Time and Wearable Sign Language Translation

AbstractFlexible strain sensors with high sensitivity and high mechanical robustness are highly desirable for their accurate and long‐term reliable service in wearable human‐machine interfaces. However, the current application of flexible strain sensors has to face a trade‐off between high sensitivity and high mechanical robustness. The most representative examples are micro/nano crack‐based sensors and serpentine meander‐based sensors. The former one typically shows high sensitivity but limited robustness, while the latter is on the contrary. Herein, ultra‐robust and sensitive flexible strain sensors are developed by crack‐like pathway customization and ingenious modulation of low/high‐resistance regions on a serpentine meander structure. The sensors show high cyclic stability (10 000 cycles), strong tolerance to harsh environments, high gauge factor (&gt;1000) comparable with that of the crack‐based sensor, and fast response time (&lt;58 ms). Finally, the sensors are integrated into a wearable sign language translation system, which is wireless, low‐cost, and lightweight. Recognition rates of over 98% are demonstrated for the translation of 21 sign languages with the assistance of machine learning. This system facilitates achieving barrier‐free communication between signers and nonsigners and offers broad application prospects in gesture interaction.

Virtual Assistant With Sign Language

This topic is all about developing a successful Sign language recognition and translation system which is helpful for people with certain disabilities who can’t use present voice enabled Virtual Assistants. Communication is important with people as it helps us understand them and gain knowledge. In case of Deaf &amp; mute people, this becomes a problem. It is solved by creating a virtual assistant built to recognize hand gestures and translate them in normal language to perform tasks ordered by the user. This certainly helps other normal people to successfully communicate with deaf &amp; mute people Without worrying about any misunderstanding at the listener’s end. There are more than 300 sign languages used by various cultural groups worldwide. In this article, we provide a technique for generating a sign language dataset using a camera, followed by the training of a TensorFlow model using transfer learning to produce a real-time sign language recognition system. Despite the small amount of the dataset, the system still performs well.

Sign Language Recognition Enabling Communication Strategies for Persons with Hearing Impairment

Sign Language is mainly used by deaf (hard hearing) and dumb people to exchange information One of the important problems that our society faces is that people with disabilities are finding it hard to cope up with the fast-growing technology. The access to communication technologies has become essential for the handicapped people. Generally deaf and dumb people use sign language for communication but they find difficulty in communicating with others who don’t understand sign language. Sign language is an expressive and natural way for communication between normal and dumb people (information majorly conveyed through the hand gesture). So, we need a translator to understand what they speak and communicate with us. The sign language translation system translates the normal sign language to speech and hence makes the communication between normal person and dumb people easier. So, the whole idea is to build a communication system that enables communications between speech-hearing impaired and a normal person. Key Word: Foundation, Communication, ENT, hearing, hearing impairment, hearing difficulties.

SIGN LANGUAGE INTERPRETER

Every average person has the ability to see, hear, and respond to their surroundings. Some unfortunate people are without this significant blessing. These people primarily include the deaf and dumb, who interact with others by using sign language. Yet, since not all common people can understand their sign language, communicating with regular people is a significant limitation for them. Also, this will make it difficult for members of the deaf and dumb communities to engage with others, especially when they want to participate in educational, social, and professional settings. The goals of this research are to create a sign language translation system for hearing or speech challenged individuals utilising deep learning and neural networks to help those who are deaf or have trouble speaking or hearing communicate with others. The model utilised in this classification assignment is a pretty simple Convolutional Neural Network implementation for the methodology (CNN). This exploits the convolutional property, which was primarily developed for the analysis of visual imagery. Three layered CNNs were trained and tested in real time using segmented RGB hand motions. With the help of a personal device, such as a laptop webcam, simple photos of the hand were used to construct the image dataset for each gesture. I was able to get training accuracy of roughly 89% and testing accuracy of 98.5% with this CNN model. Keywords : Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN), Gesture Recognition, Open CV, TensorFlow

Comparison of small sample methods for Handshape Recognition

Automatic Sign Language Translation (SLT) systems can be a great asset to improve the communication with and within deaf communities. Currently, the main issue preventing effective translation models lays in the low availability of labelled data, which hinders the use of modern deep learning models.&#x0D; SLT is a complex problem that involves many subtasks, of which handshape recognition is the most important. We compare a series of models specially tailored for small datasets to improve their performance on handshape recognition tasks. We evaluate Wide-DenseNet and few-shot Prototypical Network models with and without transfer learning, and also using Model-Agnostic Meta-Learning (MAML).&#x0D; Our findings indicate that Wide-DenseNet without transfer learning and Prototipical Networks with transfer learning provide the best results. Prototypical networks, particularly, are vastly superior when using less than 30 samples, while Wide-DenseNet achieves the best results with more samples. On the other hand, MAML does not improve performance in any scenario. These results can help to design better SLT models.

A Systematic Literature Review on Vision-Based Hand Gesture for Sign Language Translation

Deaf and hard of hearing people use sign language to communicate. People around mute and deaf people have difficulty communicating with each other if they do not understand sign language. This problem has prompted many researchers to conduct studies on sign language translation. However, there is a lack of compilation of SLR on this topic. Therefore, this paper aims to provide a thorough literature review of previous studies on sign language to text translation based on the vision method. PRISMA (Preferred Reporting Items to writing a standard Systematic Review and Meta-Analyses) is used in this systematic review. Two primary databases, Web of Science and Scopus, have been used to search for relevant articles and resources included in this systematic literature review. Based on the outcome of the systematic review of the topic, the primary studies on sign language translation systems were conducted using self-generated datasets more than public datasets. More static action sign language was studied compared to dynamic action sign language. For the type of recognition, more alphabet sign language was studied compared to digit, word, or sentence sign language. Other than that, most studies used digital cameras rather than Microsoft Kinect or a webcam. The most used classification method was Convolution Neural Network (CNN). The study is intended to guide readers and researchers for future research and knowledge enhancement in the field of sign language recognition.

Automated Arabic-Arabic sign language translation system based on 3D avatar technology

&lt;p&gt;&lt;span&gt;Arabic sign language (ArSL) is the natural language of the deaf community in Arabic countries. Deaf people have a set of difficulties due to poor services available. They have problems accessing essential information or receiving an education, communicating with other communities, and engaging in activities. Thus, a machine translation system of Arabic to ArSL has been developed using avatar technologies. Firstly, a dictionary of ArSL was constructed using eSign editor Software. The constructed dictionary has three thousand signs. It can be adopted for the translation system in which written text can be transformed into sign language. The dictionary will be available as a free resource for researchers. It is complex and time-consuming, but it is an essential step in the machine translation of whole Arabic text to ArSL with 3D animations. Secondly, the translator has been developed. It performs syntactic and morphological analysis and then applies a set of rules to translate an Arabic text into ArSL text based on the structure and grammar of ArSL. The system is evaluated according to the parallel corpus that consists of 180 sentences using the metric for evaluation of translation with explicit ordering &lt;/span&gt;&lt;span lang="IN"&gt;metric for evaluation of translation with explicit ordering&lt;/span&gt;&lt;span&gt;(METEOR) our system achieves a relative score of (86%).&lt;/span&gt;&lt;/p&gt;

Sign Language Translation Systems

Sign language, often termed “dactylology,” is a mode of communication for those who are hard of hearing. With over 2.5 billion people projected to have hearing loss by 2050, there are very few efficient real-time sign language translation (SLT) applications present today despite extensive research in the domain. The main purpose of the systematic literature review is to analyze existing research in SLT systems and obtain results that will help in building an efficient and improved SLT system. A total of 125 different research articles within the time frame of 2015–2022 were identified. The study analyzes each paper against nine main research questions. The results obtained show the unique strengths and weaknesses of the different methods used, and while the reviewed papers showed significant results, there is still room for improvement in the implementations. This systematic literature review helps in identifying suitable methods to develop an efficient SLT application, identifies research gaps in this domain, and simultaneously indicates recent trends in the field of SLT systems.