Sign Language Words Research Articles

Communication through Hands in Sign Language - A CNN Collaborative Study

A system of communication called sign language makes use of visual motions and signals. The only form of communication for the deaf and dumb community and others with hearing impairments is sign language. Understanding sign language is so much difficult for a normal person. As a result, connecting with the wider public has always been extremely difficult for the minority community. In this study, we suggested a novel deep learning-based method for identifying sign language that can help normal and deaf individuals communicate more easily. In order to identify real-time sign language, we first created a dataset with 11 sign terms. Our bespoke CNN model was trained using these sign words. Prior to the CNN model being trained, we preprocessed the dataset. Our results show that the customized CNN model can attain the greatest accuracy of 98.6%.

Novel Wearable System to Recognize Sign Language in Real Time.

The aim of this study is to develop a practical software solution for real-time recognition of sign language words using two arms. This will facilitate communication between hearing-impaired individuals and those who can hear. We are aware of several sign language recognition systems developed using different technologies, including cameras, armbands, and gloves. However, the system we propose in this study stands out for its practicality, utilizing surface electromyography (muscle activity) and inertial measurement unit (motion dynamics) data from both arms. We address the drawbacks of other methods, such as high costs, low accuracy due to ambient light and obstacles, and complex hardware requirements, which have limited their practical application. Our software can run on different operating systems using digital signal processing and machine learning methods specific to this study. For the test, we created a dataset of 80 words based on their frequency of use in daily life and performed a thorough feature extraction process. We tested the recognition performance using various classifiers and parameters and compared the results. The random forest algorithm emerged as the most successful, achieving a remarkable 99.875% accuracy, while the naïve Bayes algorithm had the lowest success rate with 87.625% accuracy. The new system promises to significantly improve communication for people with hearing disabilities and ensures seamless integration into daily life without compromising user comfort or lifestyle quality.

Picture book videos incorporating key word signing for children who require AAC and their families: design and evaluation of a multimedia-enhanced setting

Key word signing (KWS) is an unaided form of augmentative and alternative communication (AAC) and is frequently used by children with cognitive impairments and their families. Successful implementation of KWS requires a family environment that provides aided language input by modeling the signs. However, families face challenges implementing the signs in their everyday lives. KWS requires effort and sustained parental commitment. Users may also struggle with finding good learning resources and stimulating and enjoyable shared contexts for communication. Signed videos of popular children’s books may help to implement KWS and create a signing environment which exposes children and their families to KWS in meaningful ways. The aim of this study was to create videos of this type and investigate whether and how they might serve as an attractive medium of support for families’ KWS experience. Three families tested the videos. A triangulated qualitative study incorporating interviews and participant observation explored the families’ experience of using these videos as a context for shared communication. The findings suggest that picture book videos supplemented by KWS may be appropriate resources for the use of KWS in everyday family life. They serve as a child-centered activity involving KWS exposure, in which children and their families can participate joyfully and naturally.

Enhancing word signing in hearing students with reading disorders using computer-based learning trials.

Multiple-baseline-across-word-sets designs were used to determine whether a computer-based intervention would enhance accurate word signing with four participants. Each participant was a hearing college student with reading disorders. Learning trials included 3 s to observe printed words on the screen and a video model performing the sign twice (i.e., simultaneous prompting), 3 s to make the sign, 3 s to observe the same clip, and 3 s to make the sign again. For each participant and word set, no words were accurately signed during baseline. After the intervention, all four participants increased their accurate word signing across all three word sets, providing 12 demonstrations of experimental control. For each participant, accurate word signing was maintained. Application of efficient, technology-based, simultaneous prompting interventions for enhancing American Sign Language learning and future research designed to investigate causal mechanisms and optimize intervention effects are discussed.

AI Powered Sign Language Translator

The purpose of this effort is to promote equitable environments and allow people with hearing disabilities to communicate in their native language by proposing an AI-based sign language translator. We used a transformer neural network, which can analyze over 500 data points from a person's face and gestures, to translate sign language into text. The translator can expand, produce new datasets, and build models for sign language recognition thanks to our machine learning process. As a proof of concept, we developed an interpreter for emergency calls using more than 200 sign language words. The main goal is to empower people who are deaf to participate in social, political, economic, and cultural spheres of life. We see a lot of people with illnesses, including blindness, deafness, and dumbness, every day. They have trouble interacting with other people. The suggested method can translate sign language into text and voice since this study describes two-way communication between deaf, dumb, and normal individuals. Key Words: Sign Language, Inclusion, Social Development, Artificial Intelligence, Machine Learning.

A study of motivation for occupation words in Korean sign language and Chinese sign language

A study of motivation for occupation words in Korean sign language and Chinese sign language

Sign Language Detection Using Deep Learning

Abstract: Millions of citizens worldwide suffer from deaf and hard of hearing (DHH), a communication impairment that makes speaking difficult and necessitates the use of sign language. This communication gap frequently hampers access to education and opportunities for employment. Although AI-driven technologies have been studied to tackle this problem, no research has specifically looked into the intelligent and automatic translation of American sign gestures to text in low-resource languages (LRL), such as Nigerian languages. We suggest a unique end-to-end system for translating the American Sign Language, or ASL, Our framework uses the "no language left behind" translation model and the Transformer-based model for ASL-to-Text generation. converting the text generated from LRL to English. We assessed the ASL-to-Text system's performance. The people who participated were able to understand the translated text and expressed satisfaction with both the Text-to-LRL and ASL-toText models, according to a qualitative analysis of the framework. Our suggested framework shows how AI-driven technologies can promote inclusivity in sociocultural interactions and education, particularly for individuals with DHH living in low-resource environments. To bridge the gap between the non-sign language and hearing/speech impaired communities, sign language recognition is crucial. Sentence detection is more useful in real-world situations than isolated recognition of words, yet it is also more difficult since isolated signs need to be accurately identified and continuous, high-quality sign data with distinct features needs to be collected. Here, we suggest a wearable system for understanding sign language using a convolutional neural network (CNN) that combines inertial measurement units attached to the body with flexible strain sensors to detect hand postures and movement trajectories. A total of 48 frequently used ASL sign language terms were gathered and utilized to train the CNN model. This resulted in an isolated sign language word recognition accuracy of 95.85%.

Isolated Video-Based Sign Language Recognition Using a Hybrid CNN-LSTM Framework Based on Attention Mechanism

Sign language is a complex language that uses hand gestures, body movements, and facial expressions and is majorly used by the deaf community. Sign language recognition (SLR) is a popular research domain as it provides an efficient and reliable solution to bridge the communication gap between people who are hard of hearing and those with good hearing. Recognizing isolated sign language words from video is a challenging research area in computer vision. This paper proposes a hybrid SLR framework that combines a convolutional neural network (CNN) and an attention-based long-short-term memory (LSTM) neural network. We used MobileNetV2 as a backbone model due to its lightweight structure, which reduces the complexity of the model architecture for deriving meaningful features from the video frame sequence. The spatial features are fed to LSTM optimized with an attention mechanism to select the significant gesture cues from the video frames and focus on salient features from the sequential data. The proposed method is evaluated on a benchmark WLASL dataset with 100 classes based on precision, recall, F1-score, and 5-fold cross-validation metrics. Our methodology acquired an average accuracy of 84.65%. The experiment results illustrate that our model performed effectively and computationally efficiently compared to other state-of-the-art methods.

Mapping the unique neural engagement in deaf individuals during picture, word, and sign language processing: fMRI study.

Employing functional magnetic resonance imaging (fMRI) techniques, we conducted a comprehensive analysis of neural responses during sign language, picture, and word processing tasks in a cohort of 35 deaf participants and contrasted these responses with those of 35 hearing counterparts. Our voxel-based analysis unveiled distinct patterns of brain activation during language processing tasks. Deaf individuals exhibited robust bilateral activation in the superior temporal regions during sign language processing, signifying the profound neural adaptations associated with sign comprehension. Similarly, during picture processing, the deaf cohort displayed activation in the right angular, right calcarine, right middle temporal, and left angular gyrus regions, elucidating the neural dynamics engaged in visual processing tasks. Intriguingly, during word processing, the deaf group engaged the right insula and right fusiform gyrus, suggesting compensatory mechanisms at play during linguistic tasks. Notably, the control group failed to manifest additional or distinctive regions in any of the tasks when compared to the deaf cohort, underscoring the unique neural signatures within the deaf population. Multivariate Pattern Analysis (MVPA) of functional connectivity provided a more nuanced perspective on connectivity patterns across tasks. Deaf participants exhibited significant activation in a myriad of brain regions, including bilateral planum temporale (PT), postcentral gyrus, insula, and inferior frontal regions, among others. These findings underscore the intricate neural adaptations in response to auditory deprivation. Seed-based connectivity analysis, utilizing the PT as a seed region, revealed unique connectivity pattern across tasks. These connectivity dynamics provide valuable insights into the neural interplay associated with cross-modal plasticity.

Enhancing Signer-Independent Recognition of Isolated Sign Language through Advanced Deep Learning Techniques and Feature Fusion

Sign Language Recognition (SLR) systems are crucial bridges facilitating communication between deaf or hard-of-hearing individuals and the hearing world. Existing SLR technologies, while advancing, often grapple with challenges such as accurately capturing the dynamic and complex nature of sign language, which includes both manual and non-manual elements like facial expressions and body movements. These systems sometimes fall short in environments with different backgrounds or lighting conditions, hindering their practical applicability and robustness. This study introduces an innovative approach to isolated sign language word recognition using a novel deep learning model that combines the strengths of both residual three-dimensional (R3D) and temporally separated (R(2+1)D) convolutional blocks. The R3(2+1)D-SLR network model demonstrates a superior ability to capture the intricate spatial and temporal features crucial for accurate sign recognition. Our system combines data from the signer’s body, hands, and face, extracted using the R3(2+1)D-SLR model, and employs a Support Vector Machine (SVM) for classification. It demonstrates remarkable improvements in accuracy and robustness across various backgrounds by utilizing pose data over RGB data. With this pose-based approach, our proposed system achieved 94.52% and 98.53% test accuracy in signer-independent evaluations on the BosphorusSign22k-general and LSA64 datasets.

Sign Language Recognition by Image Processing

A prototype sign language interpreter that can converse in American Sign Language (ASL) is presented in this study. The visual stimulation of sign language aids in the development of speech and language. It promotes social contact, fosters the development of cognitive structures, and lessens harmful social behaviors. Early exposure to sign language supports language development in academic, social, and emotional domains. Building a user-hand gesture-only human-computer interface (HCI) is the foundation of the dependable paradigm. Using Python and OpenCV, hand gesture recognition is built using the theories of hand segmentation and the hand detection system. The training photos are labeled with different names based on what is in the database. A binary feature vector is created from each image. Convolutional neural networks (CNNs) and cameras are used to capture images from hand sign videos in real time, which are then used to carry out the feature extraction process. The algorithm generates output based on the projected maximum similarity between these features and the features of database photographs. With the aid of an application, this model provides adequate accuracy even in the absence of a constant or monochromatic background. The person's hand movements are captured on video with the camera; the video is subsequently processed and translated into text. The user hears the speech that has been transformed from the output. The suggested system recognizes letters, numbers, and certain common words in sign language. Raw photos and videos are transformed into corresponding text that can be read and comprehended by using image processing algorithms and neural networks to map the gesture to the right text in the training data. When paired with a sizable source database, this protocol will surely be highly beneficial for closing the communication gap between people who can talk and hear and those who cannot.

Characterizing language production across modalities

ABSTRACT This study investigates factors influencing lexical access in language production across modalities (signed and oral). Data from deaf and hearing signers were reanalyzed (Baus and Costa, 2015, On the temporal dynamics of sign production: An ERP study in Catalan Sign Language (LSC). Brain Research, 1609(1), 40-53. https://doi.org/10.1016/j.brainres.2015.03.013; Gimeno-Martínez and Baus, 2022, Iconicity in sign language production: Task matters. Neuropsychologia, 167, 108166. https://doi.org/10.1016/j.neuropsychologia.2022.108166) testing the influence of psycholinguistic variables and ERP mean amplitudes on signing and naming latencies. Deaf signers' signing latencies were influenced by sign iconicity in the picture signing task, and by spoken psycholinguistic variables in the word-to-sign translation task. Additionally, ERP amplitudes before response influenced signing but not translation latencies. Hearing signers' latencies, both signing and naming, were influenced by sign iconicity and word frequency, with early ERP amplitudes predicting only naming latencies. These findings highlight general and modality-specific determinants of lexical access in language production.

Spatial–temporal feature-based End-to-end Fourier network for 3D sign language recognition

Most dynamic sign word misclassifications are caused by redundant spatial–temporal (SPT) feature pruning that lacks language semantic and temporal dependencies. SPT feature recognition is one of the important aspects for the evaluation of the misclassification of dynamic sign words. The redundant pruning of SPT feature space influences the language model of sign confusion, model complexity, and SPT feature similarity. The purpose of this article is to develop a new multi-scale SPT feature-based dynamic sign word recognition approach via a low-cost feature selection method (FS) and End-to-end Fourier convolution neural network (EFCNN). Instead of a sensor fusion technique for obtaining frame position alignment, in the EFCNN, new 3D frame position and coordinates are determined using a pixel weighting and alignment function of the first and succeeding 25 spatial intensities of the 3D video changes across hand motion. The new spatial weight and the original spatial coordinates are fused and truncated in the Fourier domain. We generate the temporal dependence of the fused features. A feature selection known as the FS-EFCNN is introduced to select compact features with a preserved language meaning. Five state-of-the-art feature selection methods, namely Infinite FS (InFS), Relief FS, Fisher, MIM, ILFS, and ensemble FS-EFCNN were deployed to guide and optimize the learning performance of EFCNN. The experimental result analysis highlighted the improved results of the FS-EFCNN method with the best accuracy of 99.86%, 99.89%, and 90.69% on 3D American Sign Language, British Sign Language, and Greek Sign Language data sets, respectively.

Recognition of Wh-Question Sign Gestures in Video Streams using an Attention Driven C3D-BiLSTM Network

Human sign language is a visual and gestural means of communication used by people with hearing impairments to interact with others. It has the potential to enable interaction between individuals who struggle with verbal communication and those who lack the essential proficiency to comprehend sign language. Wh-questions constitute a substantial portion of daily sign language interactions. Automatic recognition of Wh-question words in sign language, encompassing interrogative words, has received limited attention in gesture recognition. In this context, a novel dataset named the American Question Sign Video dataset (AQSVd) has been introduced as a significant contribution. Recognizing Wh-question signs in video streams empowers individuals to effectively convey messages through hand movements and gestures, fostering inclusivity and accessibility in communication for deaf and hearing-impaired individuals. The paper proposes a Deep Convolutional-3D BiLSTM Multi-head Attention network for recognizing American Wh-Question word sign gestures in video streams. Incorporating Multi-head attention in the proposed model enhances its ability to capture intricate spatial and temporal features, essential for accurate gesture recognition. The proposed model is thoroughly evaluated across a range of datasets, such as INCLUDE50, WLASL-100, and our AQSVd dataset, achieving remarkable results. Specifically, our model demonstrated an outstanding 98.91% validation accuracy on the AQSVd dataset. The C3D-BiLSTM MHAttention model outperforms other state-of-the-art models, demonstrating its superiority in sign language recognition tasks. The proposed dataset and C3D-BiLSTM Multi-head Attention model contribute significantly to this field, offering potential benefits in education, human-robot interaction, and overall communication for the hearing-impaired community.

Chinese sign language recognition based on surface electromyography and motion information

Sign language (SL) has strong structural features. Various gestures and the complex trajectories of hand movements bring challenges to sign language recognition (SLR). Based on the inherent correlation between gesture and trajectory of SL action, SLR is organically divided into gesture-based recognition and gesture-related movement trajectory recognition. One hundred and twenty commonly used Chinese SL words involving 9 gestures and 8 movement trajectories, are selected as research and test objects. The method based on the amplitude state of surface electromyography (sEMG) signal and acceleration signal is used for vocabulary segmentation. The multi-sensor decision fusion method of coupled hidden Markov model is used to complete the recognition of SL vocabulary, and the average recognition rate is 90.41%. Experiments show that the method of sEMG signal and motion information fusion has good practicability in SLR.

Recognition of Bangladeshi Sign Language (BdSL) Words using Deep Convolutional Neural Networks (DCNNs)

In a world where effective communication is fundamental, individuals who are Deaf and Dumb (D&D) often face unique challenges due to their primary mode of communication—sign language. Despite the interpreters' invaluable roles, their lack of availability causes communication difficulties for the D&D individuals. This study explores whether the field of Human-Computer Interaction (HCI) could be a potential solution. The primary objective is to assist D&D individuals with computer applications that could act as mediators to bridge the communication gap between them and the wider hearing population. To ensure their independent communication, we propose an automated system that could detect specific Bangla Sign Language (BdSL) words, addressing a critical gap in the sign language detection and recognition literature. Our approach leverages deep learning and transfer learning principles to convert webcam-captured hand gestures into textual representations in real-time. The model's development and assessment rest upon 992 images created by the authors, categorized into ten distinct classes representing various BdSL words. Our findings show the DenseNet201 and ResNet50-V2 models achieve promising training and testing accuracies of 99% and 93%, respectively. Doi: 10.28991/ESJ-2023-07-06-019 Full Text: PDF

A wearable system for sign language recognition enabled by a convolutional neural network

Sign language recognition is of great significance to connect the hearing/speech impaired and non-sign language communities. Compared to isolated word recognition, sentence recognition is more practical in real-world scenarios, but is also more complicated because continuous, high-quality sign data with distinct features must be collected and isolated signs must be identified with high accuracy. Here, we propose a wearable sign language recognition system enabled by a convolutional neural network (CNN) that integrates stretchable strain sensors and inertial measurement units attached to the body to perceive hand postures and movement trajectories. Forty-eight Chinese sign language words commonly used in daily life were collected and used to train the CNN model, and an isolated sign language word recognition accuracy of 95.85% was achieved. For sentence-level sign language recognition, we proposed a method that combines multiple sliding windows and uses correlation analysis to improve the CNN recognition performance, achieving a correct rate of 84% for 50 sign language sentence samples, showing good extendibility.

Development of the American Sign Language Fingerspelling and Numbers Comprehension Test (ASL FaN-CT)

We describe the development and initial validation of the “ASL Fingerspelling and Number Comprehension Test” (ASL FaN-CT), a test of recognition proficiency for fingerspelled words in American Sign Language (ASL). Despite the relative frequency of fingerspelling in ASL discourse, learners commonly struggle to produce and perceive fingerspelling more than they do other facets of ASL. However, assessments of fingerspelling knowledge are highly underrepresented in the testing literature for signed languages. After first describing the construct, we describe test development, piloting, revisions, and evaluate the strength of the test’s validity argument vis-à-vis its intended interpretation and use as a screening instrument for current and future employees. The results of a pilot on 79 ASL learners provide strong evidence that the revised test is performing as intended and can be used to make accurate decisions about ASL learners’ proficiency in fingerspelling recognition. We conclude by describing the item properties observed in our current test, and our plans for continued validation and analysis with respect to a battery of tests of ASL proficiency currently in development.

Strategies for new word formation in NGT

Abstract How do new words arise in a sign language? We present an empirical study of newly formed words in Sign Language of the Netherlands (NGT). Five signers were asked to create new forms for known concepts for which word forms exist in NGT. Participants used sequential strategies for word formation, also found in spoken languages. More frequently, however, they used simultaneous strategies, some of which are unique for the visual-manual modality. We describe and discuss each strategy and focus on the most prominent of these, namely the simultaneous combination of meaningful form elements (Form-Meaning Units or FMUs). The abundance of simultaneous combinations of FMUs in our data cannot be explained in terms of concatenative morphology. We propose an account for word formation phenomena in NGT that uses the phonological segment as the template for simultaneous combinations of FMUs. This study shows that the FMUs function as basic building blocks for words and accordingly can be considered morphemes. Given the large set of FMUs in NGT and their frequent occurrence in the newly formed lexical words in our data, we argue for the acknowledgement of more morphological complexity within the lexicon of sign languages and for a re-evaluation of the relation between sign language phonology, morphology, and syntax.

Methods of teaching graphic-symbolic analysis in the literary and cultural-historical context of the 20th century (in terms of the novel ‟The Master and Margarita” by Mikhail Bulgakov)

The paper discusses the methodology for teaching graphic-symbolic analysis in the literary and cultural-historical context of the 20th century in terms of the novel ‟The Master and Margarita” by Mikhail Bulgakov. This method of analysis is an effective method of teaching oral monologue speech to foreign students. The purpose of this study is to develop the oral coherent speech of foreign students with the help of a graphical and symbolic analysis of the text of the novel ‟The Master and Margarita”. The set goal involves solving the following tasks: to help students from China understand the artistic features of the works by Mikhail Bulgakov, to acquire the skills of literary text analysis and, in general, to study Russian literature more deeply. The material of the presented study can be used in the preparation of training courses designed to teach the Russian language and literature to Chinese students, as well as in the development of special courses on the specifics of literature. The research methods were chosen in accordance with the object of study, its subject, purpose and objectives, our methodology is based on the use of practice-oriented provisions of semiotics, artistic and graphic-symbolic analysis of the text and the introduction of a new concept of ‟sign word” into the system of dictionary work. This study determines the effectiveness of using the method of teaching graphic-symbolic analysis in the educational process at different stages of teaching literature and the Russian language to foreign students. The novelty of the study lies in the analysis of materials relating to the peculiarities of the perception of Russian fiction by native speakers of the Chinese language. The theoretical significance of the work lies in the study of the specifics of Russian works and language in the aspect of Russian as a foreign language. The practical significance lies in the prevention of lexical and grammatical errors when using the Russian language and semantic errors in the analysis of Russian literature.