Indian Sign Research Articles

Deep Learning-based Bangla Sign Language Detection with an Edge Device

Sign language, often referred to as silent conversation, serves as a visual gesture-based primary communication medium for hearing-impaired individuals. People unable to speak or hear communicate with themselves and general persons with static and dynamic sign languages. To ensure that people who are deaf and unable to talk do not feel excluded from this fast-paced world, it is even more crucial that sign language is incorporated into our culture and that a cost-effective and straightforward method of detecting it be developed. With this goal in mind, an automatic Bangla sign language (BSL) detection system has been developed in this work using deep learning approaches and a Jetson Nano edge device. The deep learning models have been trained with Okkhornama, an open-source database and a custom author-curated dataset of 49 categories and 3,760 images have been used to validate these techniques. These images are preprocessed by auto-orienting and resizing into 416 × 416 pixels in the Roboflow framework. Next, we implemented Detectron2, EfficientDet-D0 with TensorFlow and the PyTorch-built YOLOv7 models. Jetson Nano, a portable and sufficiently dynamic NVIDIA microprocessor, is also utilized in this project to generate an object detection model and infer it from test images in real-time. The Dectectron2 model performed best concerning detection accuracy, with a mAP@.5 of 94.915 and an AP of 54.814. The YOLOv7 model achieved mAP@.5 accuracies in detecting various classes from 85 percent to 97 percent and mAP@.5-.95 from 41 percent to 53 percent. Finally, the YOLOv7 Tiny approach has been used in the Jetson Nano edge device for real-time Bangla sign language detection because of its lowest total training time and frame rate. The proposed system is expected to provide an effective solution for Bangladeshi people suffering from hearing disabilities and the inability to speak in a simple and affordable approach.

Factors Affecting the Uniformity of Sign Language: Perceptions of Teachers of Students with Hearing Impairment

The main purpose of this study was to examine the perception of teachers of students with hearing impairment (SWHI) about the factors affecting the uniformity of the sign language. The nature of study was quantitative. The population of the study was teachers of SWHI. Sample of the study teachers was 100 (M=52, F=48) of students with hearing impairment selected through convenient sampling techniques from different special education institutions. Survey method was used to collect data through self-developed questionnaire with reliability coefficient (α= 0.72). statistical measures were used to analyse the data. The result of the study highlight. This study used to find the variation in sign language which effect on teachers and students. There is no professional development for teachers per-service and in service. The study showed that teachers of students with hearing impairment faced many issues due to the variation of the sign language when students use different signs in the classroom and teachers also learned different signs. And the school did not pay attention on the professional development of the teachers that teachers can use same standard sign language. School and government should pay attention on the development of per-service and in-service teachers in accordance with Pakistan sign language.

Storytelling App for Child Using Natural Language Processing (NLP)

The paper introduces a mobile app that is specifically designed to enable hearing-impaired children to access stories. The app differs from traditional storytelling methods, which involve reading or listening to stories, as it converts written text into sign language images using two algorithms. The first algorithm utilizes speech APIs to convert text to speech, while the second algorithm uses natural language processing techniques from the NLTK toolkit to convert speech to sign language. The app is capable of translating english words into Indian sign language, making it an invaluable tool for caregivers, teachers, and parents of hearing-impaired children. This research project contributes to the field of assistive technology by providing a platform that caters to the specific needs of hearing-impaired children, allowing them to enjoy stories in a way that is tailored to their unique abilities. One of the unique features of the app is its ability to translate English words into Indian Sign Language, which is particularly beneficial for caregivers, teachers, and parents of hearing-impaired children in India. By offering a platform that caters to the specific needs of hearing-impaired children, this research project contributes significantly to the field of assistive technology. The app's innovative approach to storytelling provides a tailored experience that allows hearing-impaired children to enjoy stories in a way that is engaging and accessible. This app has the potential to transform the way hearing-impaired children access stories, and its contribution to the field of assistive technology is significant. Overall, this paper underscores the importance of developing technology that caters to the needs of diverse communities and highlights the possibilities that emerge when technology is designed to serve all members of society.

Sign Language Recognition and Response via Virtual Reality

The main aim of this project is to provide a voice to voiceless people with the help of Python and Unity prototype systems. Aforesaid project work focuses on the problem of gesture recognition in real time that sign language is used by the community of deaf people. The problem addressed is based on Digital image processing using color segmentation, Skin detection, Image Segmentation, Image Filtering, and Template Matching techniques. This system recognizes gestures of ISL(Indian Sign Language) including the alphabet and a subset of its words.

PakParse: Machine Translation from Text to Pakistan Sign Language for Effective Communication with Deaf People

The People around the world use different languages to communicate with each other. But there are many individuals who cant listen or speak. These people use gesture-based language for their communication which is called Sign Language. There is no universal Sign Language in world so it changes region wise. As Sign language cant be written and sign video approach is also expensive (in terms of create, store and manage), thats why sign animation through 3D avatar is best option. Recently Pakistan got its own Sign Language called Pakistan Sign Language (PSL). As a noteworthy population of deaf people live in country, there is no machine translator for PSL which can facilitate deaf in their communication and education. This work proposed a machine translator for Pakistan Sign Language called PakParse, which takes text (English & Urdu) as input, and animate corresponding signs by 3D avatar.

A multi-stack RNN-based neural machine translation model for English to Pakistan sign language translation

A multi-stack RNN-based neural machine translation model for English to Pakistan sign language translation

ISL recognition system using integrated mobile-net and transfer learning method

A sign language recognition system (SLRS) facilitates the flow of information between the signer and non-signer communities. In recent years, deep learning has become ubiquitous and has also been found in the literature of SLRS, but the system designed for real-time recognition is still lacking. This paper presents a small and efficient architecture of Mobilenetv2 integrated with transfer learning for the recognition of Indian sign language (ISL). For this, firstly pre-trained weight file of mobilenetV2 has been used for the feature extraction, and then this model has a global average pooling layer (GAP), followed by a dense layer with a softmax classifier for the classification of ISL gestures. A self-collected ISL dataset consisting of 28 isolated gestures and a publicly available ASL dataset has been used to evaluate the performance. This model’s compact architecture yielded excellent results, with a high classification accuracy of 99.3% and 99.9% for the two datasets, respectively, and low computational complexity. A real-time application prototype using GUI interface has also been designed using the proposed work to translate the gestures of ISL into English. The experimental findings show that the proposed model outperforms the state-of-art method with minimum computational and memory requirements.

Gesture-to-Text Translation Using SURF for Indian Sign Language

This research paper focuses on developing an effective gesture-to-text translation system using state-of-the-art computer vision techniques. The existing research on sign language translation has yet to utilize skin masking, edge detection, and feature extraction techniques to their full potential. Therefore, this study employs the speeded-up robust features (SURF) model for feature extraction, which is resistant to variations such as rotation, perspective scaling, and occlusion. The proposed system utilizes a bag of visual words (BoVW) model for gesture-to-text conversion. The study uses a dataset of 42,000 photographs consisting of alphabets (A–Z) and numbers (1–9), divided into 35 classes with 1200 shots per class. The pre-processing phase includes skin masking, where the RGB color space is converted to the HSV color space, and Canny edge detection is used for sharp edge detection. The SURF elements are grouped and converted to a visual language using the K-means mini-batch clustering technique. The proposed system’s performance is evaluated using several machine learning algorithms such as naïve Bayes, logistic regression, K nearest neighbors, support vector machine, and convolutional neural network. All the algorithms benefited from SURF, and the system’s accuracy is promising, ranging from 79% to 92%. This research study not only presents the development of an effective gesture-to-text translation system but also highlights the importance of using skin masking, edge detection, and feature extraction techniques to their full potential in sign language translation. The proposed system aims to bridge the communication gap between individuals who cannot speak and those who cannot understand Indian Sign Language (ISL).

Embodying Relationality and Enacting Resistance: Celluloid Mobilities in the Silent Film The Daughter Of Dawn

Abstract: “Embodying Relationality and Enacting Resistance: Celluloid Mobilities in the Silent Film The Daughter Of Dawn ” discusses how Kiowa and Comanche actors embedded their respective material cultures in the 1920 silent film The Daughter of Dawn . Filmed in the Wichita Wildlife Refuge and featuring an all-Indigenous cast, this film is a unique entry in the canon of silent Westerns produced in Oklahoma during its nascent statehood. The author illustrates how the Kiowa and Comanche cast enacted what Michelle Raheja calls “visual sovereignty,” engaging in traditional cultural practices, both on and off camera, which were under attack by settler institutions. The performance and recording of these dynamic instances of buffalo culture, traditional dance, and Plains Indian Sign Language, which the author collectively refers to as “celluloid mobilities,” constituted an embodied refusal of settler-colonial interference in Indigenous cultural continuance.

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

Hand Gesture Recognition System for Deaf and Dumb People

Deaf and dumb people communicate using language known as Sign language. There are various sign languages but comparing to other sign languages, Indian Sign Language interpretation has got least mindfulness by researchers. Alphabets in English sign language and various phrases can be recognized using the application. It deals with images of bare hands. Hand-Gesture recognition is the ability to recognize hand-gesture. The hand gesture recognition system is solution to this problem which uses the image of hand gesture and recognize the sign present in the image. These features based on gesture analysis of the hand on which image of sign is shown and it extract slant and slope information. System provides an opportunity for deaf and dumb people to communicate with normal people effortlessly. Our research focuses on recognition of English sign language for vast reach of audience. There have been several upgrades in technology and a lot of research has been done to aid the deaf and dumb people. Aiding the cause, Deep learning along with computer vision is being used to make an impact on this cause. It can help the deaf and dumb people in communicating with others as many of us don’t know what sign language is. In this sign language recognition project, we create an kotlin application, which detects numbers from 1 to 10 and also the English alphabets. The initial step of this system is to build a database of English Sign Language. Hand gesture detection is the most determining step in each hand gesture recognition system since if we get better segmented output, better recognition rates can be achieved. The proposed system also includes efficient and robust hand segmentation and tracking algorithm to achieve finer recognition rates. Various isolated words from the Standard English sign language have been recognized using a large set of samples. In proposed system, we intend to recognize some very basic elements of sign language and to translate them to text .

Multiple Languages to Sign Language Using NLTK

Sign language involves visual gestures and signs that deaf and mute people adopt as their first conversation. People other than deaf or hard of hearing can also use sign language, like, people suffering from autism, apraxia of speech, cerebral palsy or down syndrome. It entails hand gestures, non-verbal communication or physical movement, and face emotions all at the same time. It may be used by communities who have difficulty to speak, those who can hear but cannot talk, and helps normal individuals to interact with hearing impaired people. There are several groups of hearing-impaired people throughout the globe, and each community's language is unique. American Sign Language (ASL) is utilized throughout United States; British Sign Language (BSL) is a form of communication in the United Kingdom; Australian Sign Language (AUSLAN) is used in Australia; and Indian Sign Language (ISL) is used in India. ISL signs are divided into three categories: one handed, two handed, and non-manual signs. One-handed and two-handed signs are sometimes known as manual signs. Changes in body position and facial emotions provide non-manual indicators. This translator converts English text to Sign language, allowing hearing-impaired persons in India to engage with others. This application takes multiple languages as input and gives signs as information.

A Weighted Deep Ensemble for Indian Sign Language Recognition

This work concentrates on developing an Indian sign language (ISL) recognition system using a forearm-worn wearable device to assist hearing-impaired persons. A novel ensemble of convolution neural networks (CNN) is proposed for robust ISL recognition using multi-sensor data. The accuracy for classification of 50 ISL signs improved from 92.5% obtained using a single CNN to 94.2% with 10 ensemble members created using the bagging approach and soft-voting for decision aggregation. Then, the ensemble of CNNs was optimized using weighted voting, where the weights were determined using a differential evolution algorithm. This further improved the classification accuracy to 96.6% with 10 ensemble members.

A framework to recognize the sign language system for deaf and dumb using mining techniques

<span lang="EN-US">One way of communicating with the deaf is to speak sign language. The chief barrier to little Indian sign language (ISL) research was the language diversity and variations in place. It is essential to learn sign language to communicate with them. Most learning takes place in peer groups. There are very few materials available for teaching signs. Thus, signing is very challenging to learn. Fingerspelling is the first step in sign learning and is used whenever there is no appropriate sign or if the signatory is unfamiliar with it. Sign language learning tools currently available use expensive external sensors. Through this project, we will take this field further by collecting a dataset and extracting functionally helpful information used in several supervised learning methods. Our current work presents four validated fold cross results for multiple approaches. The difference from the previous work is that we used different figures for our validation set than the training set in four-fold cross-validations.</span>

Real Time Indian Sign Language Recognition using Deep LSTM Networks

Abstract: On our planet, people with speech and hearing disabilities are part of society. Communication becomes difficult when it is necessary to interact with survivors and the general public. In some races, people with disabilities practice different sign languages for communication. For people with speech and hearing disabilities, sign language is a basic means of communication in everyday life. However, a large portion of our community is unaware of the sign languages they practice, so bringing them into the mainstream is an incredible challenge. Today, computer vision-based solutions are still well received for helping the general public understand sign language. Many analysts are trying out one of his computer vision-based sign language recognition solutions, Recognition of Hand Gesture. Lately it’s been a popular area of research in various vernacular languages being used across the world. Through this research work, we propose a solution to this problem using Keypoint identification and Neural Network architecture for real time sign language recognition. This architecture uses a Long Short Term Memory architecture (LSTM) giving the best accuracy of 91.5% in prediction of words as well as phrases of Indian Sign Language

Multi-level Taxonomy Review for Sign Language Recognition: Emphasis on Indian Sign Language

With the phenomenal increase in image and video databases, there is an increase in the human-computer interaction that recognizes Sign Language. Exchanging information using different gestures between two people is sign language, known as non-verbal communication. Sign language recognition is already done in various languages; however, for Indian Sign Language, there is no adequate amount of work done. This article presents a review on sign language recognition for multiple languages. Data acquisition methods have been over-viewed in four ways (a) Glove-based, (b) Kinect-based, (c) Leap motion controller, and (d) Vision-based. Some of them have pros and cons that have also been discussed for every data acquisition method. Applications of sign language recognition are also discussed. Furthermore, this review also creates a coherent taxonomy to represent the modern research divided into three levels: Level 1 Elementary level (Recognition of sign characters), Level 2 Advanced level (Recognition of sign words), and Level 3 Professional level (Sentence interpretation). The available challenges and issues for each level are also explored in this research to provide valuable perceptions into technological environments. Various publicly available datasets for different sign languages are also discussed. An efficient review of this article shows that the significant exploration of communication via sign acknowledgment has been performed on static, dynamic, isolated, and continuous gestures using various acquisition methods. Comprehensively, the hope is that this study will enable readers to learn new pathways and gain knowledge to carry out further research work in the domain related to sign language recognition.

LiST: A Lightweight Framework for Continuous Indian Sign Language Translation

Sign language is a natural, structured, and complete form of communication to exchange information. Non-verbal communicators, also referred to as hearing impaired and hard of hearing (HI&HH), consider sign language an elemental mode of communication to convey information. As this language is less familiar among a large percentage of the human population, an automatic sign language translator that can act as an interpreter and remove the language barrier is mandatory. The advent of deep learning has resulted in the availability of several sign language translation (SLT) models. However, SLT models are complex, resulting in increased latency in language translation. Furthermore, SLT models consider only hand gestures for further processing, which might lead to the misinterpretation of ambiguous sign language words. In this paper, we propose a lightweight SLT framework, LiST (Lightweight Sign language Translation), that simultaneously considers multiple modalities, such as hand gestures, facial expressions, and hand orientation, from an Indian sign video. The Inception V3 architecture handles the features associated with different signer modalities, resulting in the generation of a feature map, which is processed by a two-layered (long short-term memory) (LSTM) architecture. This sequence helps in sentence-by-sentence recognition and in the translation of sign language into text and audio. The model was tested with continuous Indian Sign Language (ISL) sentences taken from the INCLUDE dataset. The experimental results show that the LiST framework achieved a high translation accuracy of 91.2% and a prediction accuracy of 95.9% while maintaining a low word-level translation error compared to other existing models.

A Comprehensive Review on Indian Sign Language Recognition System using Vision based Approaches

A Comprehensive Review on Indian Sign Language Recognition System using Vision based Approaches

An Efficient Approach for Interpretation of Indian Sign Language using Machine Learning

Non-verbal communication involves the usage of Sign Language. The sign language is used by people with hearing / speech disabilities to express their thoughts and feelings. But normally, people find it difficult to understand the hand gestures of the specially challenged people as they do not know the meaning of the sign language gestures. Usually, a translator is needed when a speech / hearing impaired person wants to communicate with an ordinary person and vice versa. In order to enable the specially challenged people to effectively communicate with the people around them, a system that translates the Indian Sign Language (ISL) hand gestures of numbers (1-9), English alphabets (A-Z) and a few English words to understandable text and vice versa has been proposed in this paper. This is done using image processing techniques and Machine Learning algorithms. Different neural network classifiers are developed, tested and validated for their performance in gesture recognition and the most efficient classifier is identified.

Enhancing Collaborative Interaction with the Augmentation of Sign Language for the Vocally Challenged

As per Census 2011, in India, there were 26.8 million differently abled people, out of which more than 25%of the people faced difficulty in vocal communication. They use Indian Sign Language (ISL) to communicate with others. The proposed solution is developing a sensor-based Hand Gesture Recognition (HGR) wearable device capable of translating and conveying messages from the vocally challenged community. The proposed method involves designing the hand glove by integrating flex and Inertial Measurement Unit (IMU) sensors within the HGR wearable device, wherein the hand and finger movements are captured as gestures. They are mapped to the ISL dictionary using machine learning techniques that learn the spatio-temporal variations in the gestures for classification. The novelty of the work is to enhance the capacity of HGR by extracting the spatio-temporal variations of the individual’s gestures and adapt it to their dynamics with aging and context factors by proposing Dynamic Spatio-temporal Warping (DSTW) technique along with long short term memory based learning model. Using the sequence of identified gestures along with their ISL mapping, grammatically correct sentences are constructed using transformer-based Natural Language Processing (NLP) models. Later, the sentences are conveyed to the user through a suitable communicable media, such as, text-to-voice, text-image, etc. Implementation of the proposed HGR device along with the Bidirectional Long-Short Memory (BiLSTM) and DSTW techniques is carried out to evaluate the performance with respect to accuracy, precision and reliability for gesture recognition. Experiments were carried out to capture the varied gestures and their recognition, and an accuracy of 98.91% was observed.