Mutism Research Articles

Sign Tone: A Deep Learning-Based Deaf Companion System for Two Way Communication Between Deaf and Non-Deaf Individuals

Communication is essential to express and receive information, knowledge, ideas, and views among people, but it has been quite a while to be an obstruction for people with hearing and mute disabilities. Sign language is one method of communicating with deaf people. Though there is sign language to communicate with non-sign people it is difficult for everyone to interpret and understand. The performance of existing sign language recognition approaches is typically limited. Developing an assistive device that will translate the sign language to a readable format will help the deaf-mutes to communicate with ease to the common people. Recent advancements in the development of deep learning, deep neural networks, especially Temporal convolutional networks (TCNs) have provided solutions to the communication of deaf and mute individuals. In this project, the main objective is to design Deaf Companion System for that to develop SignNet Model to provide two-way communication of deaf individuals and to implement an automatic speaking system for deaf and mute people. It provides two-way communication for all classes of people (deaf-and-mute, hard of hearing, visually impaired, and non-signers) and can be scaled commercially. The proposed system, consists of three modules; the sign recognition module (SRM) that recognizes the signs of a deaf individual using TCN, the speech recognition using Hidden Marko Model and synthesis module (SRSM) that processes the speech of a non-deaf individual and converts it to text, and an Avatar module (AM) to generate and perform the corresponding sign of the non-deaf speech, which were integrated into the sign translation companion system called deaf companion system to facilitate the communication from the deaf to the hearing and vice versa. The proposed model is trained on Indian Sign Language. Then developed a web-based user interface to deploy SignNet Model for ease of use. Experimental results on MNIST sign language recognition datasets validate the superiority of the proposed framework. The TCN model gives an accuracy of 98.5%..

Implementation on FingerScripter: Converting Fingerspelling into Text and Speech

Abstract: Communication might seem a simple act for normal people, but for the ones who are suffering from physical or psychological disability find it very difficult for communicating with others. This communication barrier needs to be removed for which there are various techniques that are being used, out of which some are verbal, non-verbal, visual and written. For mute and deaf people these communication barrier is a major problem which can adversely affect their lives adding to their already challenging lives. For this sign languages are the best possible option, as they bridge the gap between communication of these deaf and mute people with normal people. We have created a machine learning model in order to solve this issue which will identify these signs and interpret them to detect which alphabet is signified. The model will be detecting the American Sign Language (ASL) which is one of the most widely used sign language and a type of fingerspelling. The model will form words out of the identified alphabets which will also be then converted into speech. The key algorithm which will be used is a special type of Convolutional Neural Network (CNN) for image recognition. All the documentation regarding the model along with its working is provided in this paper. With high precision and accuracy the model aims to ease the communication of deaf and mute people.

All Country Sign Language Recognition System to Help Deaf and Mute People

With the use of contemporary technology, the "All Country Sign Language Recognition System to Help Deaf and Mute People" project seeks to provide a strong solution that will close communication barriers between the deaf and mute population and the general public. The project aims to use machine learning techniques and computer vision techniques to properly understand and convert sign language motions into generally used languages, with a concentration on Indian sign language. Deaf and mute people will benefit from increased accessibility and inclusion in a variety of spheres of everyday life, including social interactions, work, and education, by virtue of the implementation of this system. The initiative helps to create a more inclusive society in addition to addressing the urgent demand for efficient communication tools for the deaf and mute communities.In this people can interact using different country sign languages like ISL, ASL, BSL, etc.

The use of TensorFlow Action Recognition as the Main Component in Making a Sign Language Translator Speaker for Speech-Impaired People

Due to communication barriers, deaf and mute students are separated from their friends, families and communities as their schools do not offer sign language instruction. Consequently, this cluster of people may feel excluded from their communities, depriving them the chance of living a normal life that is free from discrimination. The objective of this quantitative experimental study is to use TensorFlow Action Recognition as the main component in making a Sign Language Translator Speaker for Speech-Impaired People. Based on the results, the device can successfully translate sign languages with an average of 5.91 seconds, and translate three signs per 30 seconds. Also, it was found that it can detect distances up to four meters. The study manifested that the device provides the service of breaking past the communication barriers to the speech- impaired and hearing-impaired individuals, which advocates and facilitates effective communication while fostering inclusivity. These results affirmed that it is feasible to make a Sign Language Translator Speaker with the use of TensorFlow Action Recognition. Thus, this Sign Language Speaker device offers the best services for deaf and mute people Qatar and all around the world, as the struggles of hearing and speech- impaired people can be alleviated.

Smart Hand Gloves for Deaf and Dumb People

Abstract: Approximately 2.78% of the population in our nation is nonverbal. They communicate with people by using their hands and facial gestures. As is well known, those who are silent are mute and those who are not are unaware of sign language, which is a way for silent individuals to communicate. Thus, we have put forth a brand-new method known as “The SPEAKING SYSTEM TO THE MUTE PEOPLE”. Thus, this system will be helpful in solving the issue. The accelerometer sensors form the foundation of the system. Mute people believe that every deed has a purpose. The accelerometer sensors accelerate with each action and provide a signal to the Arduino. The speech signal is created by the Arduino by synchronizing its motion with that of the database. The Guide Me app is being used as the system's output. Since gestures are consistent with people's communication habits, a lot of study has been done on gesture detection using a vision-based method.

Indian Sign Language Recognition using transfer learning with Efficient Net

ABSTRACT Automatic conversion of sign language into text or speech can be really useful in communicating among deaf or mute people with no sign language knowledge. The present need is to create an automated system that can translate ISL signals into regular text and vice versa. It is advantageous for both groups to communicate their needs to one another while using publicly accessible services, such as banking, traffic signaling, and ticketing. In this manuscript, Indian Sign Language Recognition using Transfer Learning with Efficient Net (ISLR-EfficientNetB0) is proposed for Indian sign language recognition. Initially, the input images are gathered via the Indian Sign Language (ISL) dataset. The input image is pre-processing by Fairness-aware Collaborative Filtering (FACF) method. In pre-processing stage noise are removed from input images and image resizing is done. Then, Efficient Net B0 is employed to identify ISL. The proposed model is implemented on Python. Then the efficiency of the proposed approach is compared with existing models. The accuracy, precision, specificity, sensitivity, F1-score, computational time, error rate and RoC of the suggested approaches are all assessed. The proposed ISLR-EfficientNetB0 method attains 31.42%, 26.04%, and 20.45% higher accuracy, 19.24%, 26.35%, and 24.02% higher precision, 15.84%, 18.91%, and 28.16% higher specificity and 17.06%, 21.36%, and 32.47% higher sensitivity compared with existing methods, such as Indian sign language recognition under machine learning methods (ISLR-KNN), ISL recognition scheme utilizing SURF with SVM and CNN (ISLR-SVM-CNN) and Benchmarking deep neural network methods for ISL recognition (ISLR-VGG16) respectively.

Bengali sign language translator with location tracking system

Designing an embedded system to convert sign language to sound forms to communicate with the outside world can be a challenging yet rewarding project, especially for mute people. To convey a speaker's thought through sign language, hand shapes, hand orientation and movement, and facial expressions must be combined concurrently. This research is intended to design a system that translates sign language into sound forms to establish communication with the outside world for people who are deaf, those who can hear but cannot physically speak, or have trouble with spoken languages due to some other disabilities. They can thus receive prompt assistance and stay out of uncomfortable circumstances. Additionally, this system incorporates a tracking system that uses a global system for mobile communications (GSM)/ global positioning system (GPS) module to locate a person using a tracking device and send the location to previously saved emergency contact numbers so that someone nearby can quickly locate and assist the person. Typically, each nation has its own native sign language. This project will create a few essential and typical sentences and phrases in Bengali.

Machine Learning Based Customized Solution for Deafness and Mute People

Individuals with hearing or speech impairments often possess a higher degree of familiarity with sign language due to its widespread usage. Consequently, a noticeable communication divide exists between those with such impairments and the general populace. A pivotal solution in bridging this gap involves employing human sign language interpreters. Regrettably, the scarcity of sign language interpreters worldwide in comparison to the number of individuals with hearing or speech impairments has resulted in certain individuals being unable to bear the cost of a human interpreter each time they engage in conversation. To address this issue, it is imperative to automate communication in a manner that liberates the deaf community from dependence on human translators. This article centers on the development of an application capable of real-time conversion between American Sign Language and text, along with supplementary functionalities aimed at dismantling the communication barriers faced by individuals with hearing or speech challenges when interacting with the broader public. the primary objective of this application is to accurately perceive and interpret the user's sign language. The initial stage involves training the system to recognize and interpret signs utilizing algorithms for object recognition and motion tracking. For this purpose, a convolutional neural network model was trained employing a meticulously curated American Sign Language dataset. Subsequently, the recognized signs are translated into English, forming grammatically sound words. Incorporated within the system are a language translator and a virtual sign keyboard, augmenting its capabilities. Moreover, a text-to-text transformer, structured on an encoder-decoder framework, is harnessed to identify grammatical inaccuracies and generate coherent phrases. For a comprehensive understanding of the process, detailed elaboration will follow in subsequent sections.

(ICIC) Sign Suit: A Technological Solution for the Improvement of the Facilitated Communication of the Deaf and Mute People

(ICIC) Sign Suit: A Technological Solution for the Improvement of the Facilitated Communication of the Deaf and Mute People

Real-Time Sign Language Fingerspelling Recognition using Convolutional Neural Network

: Sign language allows mute people to communicate, problem occurs when a conversationalist fails to understand it. Despite efforts to address this problem, an effective solution is not yet found. In this work, Convolutional Neural Network (CNN) was trained on two different datasets separately- binary and Red Blue Green )RGB(, each contains 25,900 images of Nigerian Sign Language. A deep neural pre-trained module was used to detect hand gestures in the video feed which tackled the issue of complex backgrounds, also showed excellent detection in dimly lit areas. The accuracies of (98.95%, 76%) and (98.87%, 98.85%) were obtained respectively on the training and the validation sets. The real time system developed implemented both models as a single system which makes it a unique one.

A novel communication system based on sign language recognition and voice conversion for differently abled person

Sign language is a form of communication that helps deaf and mute people interact with those who can hear. In the country, around 2.78% of people are unable to speak, i.e., are mute or deaf. Their primary means of communication is through hand motions and gestures. The paper proposes a new technique called the "artificial speaking mouth" for individuals with speech impairments. Humans connect with each other through thoughts and ideas, but some people lack the ability to speak. For them, sign language is the only means of communication. Nowadays, technology has reduced this communication gap with systems that translate sign language into speech. Sign Language Recognition (SLR) and gesture-based control are two major applications used for hand gesture recognition technologies. On the other hand, the controller converts sign language into text and speech, using text-to-speech conversion and analog-to-digital conversion.

Real-Time Sign Language Fingerspelling Recognition using Convolutional Neural Network

Sign Language allows mute people to communicate, problem occurs when a conversationalist fails to understand it. Despite efforts to address this problem, an effective solution is not yet found. In this work, Convolutional Neural Network was trained on two different datasets separately- Binary and Red Blue Green )RGB(, each contains 25,900 images of Nigerian Sign Language. A Deep Neural pre-trained module was used to detect hand gestures in the video feed which tackled the issue of complex backgrounds, also showed excellent detection in dimly lit areas. The accuracies of (98.95%, 76%) and (98.87%, 98.85%) were obtained respectively on the training and the validation sets. The real time system developed implemented both models as a single system which makes it a unique one.

Assistive Magnetic Skin System for Speech Reconstruction: An Empowering Technology for Aphonic Individuals

Individuals with a voice disorder or speech sound disorder (SSD) (i.e., aphonic or mute people) can have any combination of difficulties with perception, articulation/motor production, and phonotactics, which may impact their speech intelligibility and acceptability, thus finding challenging to communicate with the public. As a result, many individuals suffer from frustration, isolation, and depression. Natural verbal communication for SSD patients is now more feasible than ever thanks to advancements in wearable artificial skins and machine learning. This article presents an assistive magnetic skin system for speech reconstruction (AM2S‐SR), which enables SSD‐afflicted people to communicate with their mouths. Using magnetic field sensors integrated into Magnetphones, the system reads the movement of the mouth by tracking the movement of magnetic skin patches attached next to the bottom lip. The measured magnetic field signals data are then processed using a Fine K‐Nearest Neighbor machine learning classifier. The classified data are then exported verbally on speakers, or visually on a display. AM2S‐SR successfully identifies the full English alphabets with average success rate of 94.96%, thus enabling SSD people to talk using the mouth and have a more natural conversation with others.

Klasifikasi Abjad SIBI (Sistem Bahasa Isyarat Indonesia) menggunakan Mediapipe dengan Metode Deep Learning

General public knowledge in Indonesia regarding the Indonesian Sign Language System (SIBI) is quite low. This can prevent deaf and mute people from doing activities in public facilities. In this paper, we propose an alphabetical classification of SIBI sign language using the mediapipe and the Deep Learning method to help deaf and mute people communicate with the public. The methodology of this paper collects a dataset in the form of image data from the hand patterns of each SIBI sign language alphabet by combining a webcam with the help of the open-cv library to retrieve image data. With the mediapipe, the data is extracted from the coordinates of the landmarks in his hand and then normalized for each coordinate. Then the model is trained with a fully connected layer deep learning algorithm. The experimental results obtained an accuracy of 94.32% and a loss of 15.17% on the training data, an accuracy of 93.52% and a loss of 18.91% on the validation data, and an accuracy of 93.94% on the test data. These results show that the mediapipe and the deep learning fully connected layer algorithm successfully detect the right and left hands, and coordinates of landmarks according to the SIBI sign language alphabet correctly

Hast Mudra: Hand Sign Gesture Recognition Using LSTM

Even though using the most natural way of communication is sign language, deaf and mute people find it challenging to socialize. A language barrier is erected between regular people and D&M individuals due to the structure of sign language, which is distinct from text. They converse by using vision-based communication as a result. The gestures can be easily understood by others if there is a standard interface that transforms sign language to visible text. As a result, R&D has been done on a vision-based interface system that will allow D&M persons to communicate without understanding one another's languages. In this project, first gathered and acquired data and created a dataset, after which extracting useful data from the images. Keywords After verification and trained data and model using the (LSTM) algorithm, TensorFlow, and Keras technology, classified the gestures according to alphabet. Using our own dataset, this system achieved an accuracy of around 86.75% in an experimental test. system uses the (LSTM) algorithm to process images and data.

Object Localization and Detecting Alphabet in Sign Language BISINDO Using Convolution Neural Network

The BISINDO sign language is used to help deaf and mute people communicate with other people. However, not everyone is able to understand the meaning of this sign language. A system that implements artificial intelligence methods is created to solve this problem. The system uses a Convolution Neural Network algorithm with object localization techniques to detect and classify the alphabet in each form of the BISINDO finger signal. The Region Convolution Neural Network (RCNN) algorithm is used to process object localization and the CNN algorithm will perform classification process. This system is trained using 64 training data and tested using 16 test data for each type of alphabet. The results of the system testing that have been carried out are able to provide excellent accuracy values, which are above 90 percent for a training epoch of at least 50. These results produce an accuracy of 90.10% and 97.33% respectively.

Behavioural Analysis of Deaf and Mute People Using Gesture Detection

Deaf and mute people have unique communication and social challenges that make it hard to express their thoughts, needs, and ideas. Understanding people's behavior is more important to protect them and help them integrate into society. This study discusses the critical need for behavioral analysis on deaf and mute people and introduces the Automatic Behavioral Analysis Employing Gesture Detection Framework (ABA-GDF). Gesture detection technology has gained popularity recently. This emphasis may be due to its ability to overcome communication hurdles and illuminate nonverbal communication. Current methods have various challenges, including limited accuracy and adaptability. The ABA-GDF architecture comprises three phases: dataset collection, modeling, and deployment. The data collection technique includes hand signals used by deaf and quiet people. The material is then processed to partition and normalize the hand area for consistent analysis. During Modelling, feature descriptor attributes are developed to extract relevant motion information. A classifier learns and predicts using the feature vectors, enabling the framework to recognize and interpret motions and actions. Large-scale simulations of ABA-GDF showed promising results. The ABA-GDF framework achieved 92% gesture recognition accuracy on the dataset. The system's robustness is demonstrated by its capacity to understand non-verbal messages. The research showed a 15% reduction in false positives compared to earlier methods, demonstrating its real-world usefulness.

Conversion of Sign Language to Text

Abstract: Sign language is a form of communication that uses hand sign and gestures to convey meaning. we present a new approach to converting sign language into text format. Our system is designed to enable deaf and mute people to communicate with others in a more accessible and convenient way. The proposed method uses computer vision and deep learning methods to recognize hand gestures and translate them into appropriate text. The system was built using a combination of key point detection using MediaPipe, data pre-processing, label, feature generation and LSTM neural network training. This work has the potential to significantly improve communication for deaf and dumb individuals and reduce the barriers to communication with the rest of the world. The system uses key point detection algorithms such as MediaPipe to identify hand gestures and a Lstm model to translate them into corresponding text output. The data collected from the sign language is pre-processed and then used to train an LSTM neural network to accurately recognize gestures and produce text output. This method of conversion not only helps deaf and hard of hearing individuals communicate with the hearing population, but also serves as an assistive tool for individuals who are trying to learn sign language. Overall, the proposed solution has the potential to greatly improve communication and reduce barriers for deaf and hard of hearing individuals

Virtual Assistant With Sign Language

This topic is all about developing a successful Sign language recognition and translation system which is helpful for people with certain disabilities who can’t use present voice enabled Virtual Assistants. Communication is important with people as it helps us understand them and gain knowledge. In case of Deaf & mute people, this becomes a problem. It is solved by creating a virtual assistant built to recognize hand gestures and translate them in normal language to perform tasks ordered by the user. This certainly helps other normal people to successfully communicate with deaf & mute people Without worrying about any misunderstanding at the listener’s end. There are more than 300 sign languages used by various cultural groups worldwide. In this article, we provide a technique for generating a sign language dataset using a camera, followed by the training of a TensorFlow model using transfer learning to produce a real-time sign language recognition system. Despite the small amount of the dataset, the system still performs well.

Sign Language Recognition using Deep Learning through LSTM and CNN

This study presents the application of using deep learning to detect, recognize and translate sign language. Understanding sign language is crucial for communication between the deaf and mute people and the general society. This helps sign language users to easily communicate with others, thus eliminating the differences between both parties. The objectives of this thesis are to extract features from the dataset for sign language recognition model and the formulation of deep learning models and the classification performance to carry out the sign language recognition. First, we develop methodology for an efficient recognition of sign language. Next is to develop multiple system using three different model which is LSTM, CNN and YOLOv5 and compare the real time test result to choose the best model with the highest accuracy. We used same datasets for all algorithms to determine the best algorithm. The YOLOv5 has achieved the highest accuracy of 97% followed by LSTM and CNN with 94% and 66.67%.