Sign Language Data Research Articles

Cross-modal knowledge distillation for continuous sign language recognition

Continuous Sign Language Recognition (CSLR) is a task which converts a sign language video into a gloss sequence. The existing deep learning based sign language recognition methods usually rely on large-scale training data and rich supervised information. However, current sign language datasets are limited, and they are only annotated at sentence-level rather than frame-level. Inadequate supervision of sign language data poses a serious challenge for sign language recognition, which may result in insufficient training of sign language recognition models. To address above problems, we propose a cross-modal knowledge distillation method for continuous sign language recognition, which contains two teacher models and one student model. One of the teacher models is the Sign2Text dialogue teacher model, which takes a sign language video and a dialogue sentence as input and outputs the sign language recognition result. The other teacher model is the Text2Gloss translation teacher model, which targets to translate a text sentence into a gloss sequence. Both teacher models can provide information-rich soft labels to assist the training of the student model, which is a general sign language recognition model. We conduct extensive experiments on multiple commonly used sign language datasets, i.e., PHOENIX 2014T, CSL-Daily and QSL, the results show that the proposed cross-modal knowledge distillation method can effectively improve the sign language recognition accuracy by transferring multi-modal information from teacher models to the student model. Code is available at https://github.com/glq-1992/cross-modal-knowledge-distillation_new.

Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model.

Sign language is an essential means of communication for individuals with hearing disabilities. However, there is a significant shortage of sign language interpreters in some languages, especially in Saudi Arabia. This shortage results in a large proportion of the hearing-impaired population being deprived of services, especially in public places. This paper aims to address this gap in accessibility by leveraging technology to develop systems capable of recognizing Arabic Sign Language (ArSL) using deep learning techniques. In this paper, we propose a hybrid model to capture the spatio-temporal aspects of sign language (i.e., letters and words). The hybrid model consists of a Convolutional Neural Network (CNN) classifier to extract spatial features from sign language data and a Long Short-Term Memory (LSTM) classifier to extract spatial and temporal characteristics to handle sequential data (i.e., hand movements). To demonstrate the feasibility of our proposed hybrid model, we created a dataset of 20 different words, resulting in 4000 images for ArSL: 10 static gesture words and 500 videos for 10 dynamic gesture words. Our proposed hybrid model demonstrates promising performance, with the CNN and LSTM classifiers achieving accuracy rates of 94.40% and 82.70%, respectively. These results indicate that our approach can significantly enhance communication accessibility for the hearing-impaired community in Saudi Arabia. Thus, this paper represents a major step toward promoting inclusivity and improving the quality of life for the hearing impaired.

"Human Hand Gesture Recognition Based Feature Extraction and Classification Using Sign Language Data"

"Human Hand Gesture Recognition Based Feature Extraction and Classification Using Sign Language Data"

Spatial–temporal feature-based End-to-end Fourier network for 3D sign language recognition

Most dynamic sign word misclassifications are caused by redundant spatial–temporal (SPT) feature pruning that lacks language semantic and temporal dependencies. SPT feature recognition is one of the important aspects for the evaluation of the misclassification of dynamic sign words. The redundant pruning of SPT feature space influences the language model of sign confusion, model complexity, and SPT feature similarity. The purpose of this article is to develop a new multi-scale SPT feature-based dynamic sign word recognition approach via a low-cost feature selection method (FS) and End-to-end Fourier convolution neural network (EFCNN). Instead of a sensor fusion technique for obtaining frame position alignment, in the EFCNN, new 3D frame position and coordinates are determined using a pixel weighting and alignment function of the first and succeeding 25 spatial intensities of the 3D video changes across hand motion. The new spatial weight and the original spatial coordinates are fused and truncated in the Fourier domain. We generate the temporal dependence of the fused features. A feature selection known as the FS-EFCNN is introduced to select compact features with a preserved language meaning. Five state-of-the-art feature selection methods, namely Infinite FS (InFS), Relief FS, Fisher, MIM, ILFS, and ensemble FS-EFCNN were deployed to guide and optimize the learning performance of EFCNN. The experimental result analysis highlighted the improved results of the FS-EFCNN method with the best accuracy of 99.86%, 99.89%, and 90.69% on 3D American Sign Language, British Sign Language, and Greek Sign Language data sets, respectively.

SignIt! An Android game for sign bilingual play that collects labelled sign language data

The Deaf or Hard-of-Hearing (DHH) community constitutes over 430 million people globally, with about 70 million of them using sign language as their primary means of communication. India has around 63 million DHH individuals. The DHH community in India faces several challenges, particularly in learning sign language and English, due to delayed diagnosis, stigma, oralism, and a diversity of languages. Digital games for spoken and sign language learning have gained popularity due to their advantages over traditional language learning methods, such as enhanced engagement and socialization, driving increased research and adoption over traditional methods. Moreover, the development of robust machine learning models for sign language recognition, which could significantly improve access for signers, is currently impeded by the scarcity of labelled sign language data. To address these challenges, we collaborated with NISH, an academic institution for the DHH community and developed SignIt!, an accessible and inclusive quiz platform that facilitates the learning of English and Indian Sign Language (ISL) with a secondary goal of data collection. To assess the game's usability, we conducted a study with 20 members of the DHH community, followed by interviewing 15 participants. Overall, our participants answered 2160 quiz questions and created 210 questions. The quiz creation resulted in the collection of three hours of labelled real-world sign language data. The interviews revealed novel insights, such as a preference for playing competitively with friends, empowerment by their agency to be content creators, and early signs of learning English, sign language, and quiz content by playing and creating quizzes. We plan to open-source and release SignIt! to increase its adoption among diverse DHH communities.

Violence against Deaf Women from an Intersectional Framework in Mexico

This research aimed to analyse the association between violence with linguistic knowledge in Mexican Sign Language (LSM) and sociodemographic data of Mexican Deaf women. To collect data from an intersectional framework, this study applied an opportunistic and chain sampling of 46 Deaf women from different areas of Mexico. Methods included in-person and remote interviews. Forty-one per cent of women indicated that they had been victims of abuse by their partner. In comparison, 73.9% of women had suffered some abuse in schools, and 39% of women had been victims of sexual abuse. The analyses with Spearman bivariate correlation showed that as the LSM learning age of the Deaf women increased, the levels of labour abuse (rs = .50, p < .0001), violence in a bank (rs = . 51, p < .0001), violence in school (rs = .56, p < .0001), and violence by an interpreter (rs = .45, p < .01) increased too. Results also revealed that a more excellent LSM family knowledge of LSM is moderately related to less labour abuse (rs = -.53, p < .0001), violence in a bank (rs = -.45, p < .01), violence in school (rs = -.43, p < .01), violence by an interpreter (rs = -.39, p < .01), and with lower magnitude to institutional violence (rs = -.37, p < .05).

Best practices for sign language technology research

AbstractResearch on sign language technology (SLT) has steadily increased in recent decades, and yet, common mistakes and pitfalls have significantly hindered progress in the field. The purpose of this paper is to examine some of the most prominent issues and suggest practical steps to overcome them, outlining the best practices to consider when conducting SLT research. These practices cluster around the five following issues: (1) knowledge of the specific sign language at the centre of the research and of sign languages more generally; (2) involving deaf people at the centre of research, including researchers who are themselves deaf; (3) motivations of the researcher and the relationship to the views of the sign language community; (4) what sign language data needs to be considered; (5) full recognition of the challenges posed by such research.

Dataglove for Sign Language Recognition of People with Hearing and Speech Impairment via Wearable Inertial Sensors.

Finding ways to enable seamless communication between deaf and able-bodied individuals has been a challenging and pressing issue. This paper proposes a solution to this problem by designing a low-cost data glove that utilizes multiple inertial sensors with the purpose of achieving efficient and accurate sign language recognition. In this study, four machine learning models-decision tree (DT), support vector machine (SVM), K-nearest neighbor method (KNN), and random forest (RF)-were employed to recognize 20 different types of dynamic sign language data used by deaf individuals. Additionally, a proposed attention-based mechanism of long and short-term memory neural networks (Attention-BiLSTM) was utilized in the process. Furthermore, this study verifies the impact of the number and position of data glove nodes on the accuracy of recognizing complex dynamic sign language. Finally, the proposed method is compared with existing state-of-the-art algorithms using nine public datasets. The results indicate that both the Attention-BiLSTM and RF algorithms have the highest performance in recognizing the twenty dynamic sign language gestures, with an accuracy of 98.85% and 97.58%, respectively. This provides evidence for the feasibility of our proposed data glove and recognition methods. This study may serve as a valuable reference for the development of wearable sign language recognition devices and promote easier communication between deaf and able-bodied individuals.

Nominal plurals in Sign Language of the Netherlands: Accounting for allomorphy and variation

In both signed and spoken languages, reduplication is a common process in the formation of morphologically complex structures, expressing, e.g., plurality and certain aspectual meanings. A framework in which spoken language reduplication has been formalized frequently is Optimality Theory (OT). While an important attribute of OT-constraints is their universality, to date, the question to what extent such constraints are modality-independent, and thus work for sign language reduplication as well, remains largely unanswered. In the present study, we offer the first OT-formalization of plural reduplication in Sign Language of the Netherlands (NGT). The NGT-data reveal that this language features different plural allomorphs, the choice of which depends on phonological properties of the base noun. However, we also identify variation, e.g., all noun types allow for zero marking.In our formalization, we aim to introduce constraints that are maximally modality-independent, using constraint types that have previously been proposed for spoken language reduplication. Our formalization is the first to take into account base-reduplicant faithfulness for a sign language, and also the first to account for variation in sign language data by employing stochastic OT, whereby some noise is added to the ranking value of each constraint at evaluation time. Evaluating the modality-(in)dependence of our proposed account suggests that the types of constraints we employ as well as the evaluation in the spirit of stochastic OT are not specific to a modality, while the featural implementation is inevitably modality-dependent.

Multi-Stream General and Graph-Based Deep Neural Networks for Skeleton-Based Sign Language Recognition

Sign language recognition (SLR) aims to bridge speech-impaired and general communities by recognizing signs from given videos. However, due to the complex background, light illumination, and subject structures in videos, researchers still face challenges in developing effective SLR systems. Many researchers have recently sought to develop skeleton-based sign language recognition systems to overcome the subject and background variation in hand gesture sign videos. However, skeleton-based SLR is still under exploration, mainly due to a lack of information and hand key point annotations. More recently, researchers have included body and face information along with hand gesture information for SLR; however, the obtained performance accuracy and generalizability properties remain unsatisfactory. In this paper, we propose a multi-stream graph-based deep neural network (SL-GDN) for a skeleton-based SLR system in order to overcome the above-mentioned problems. The main purpose of the proposed SL-GDN approach is to improve the generalizability and performance accuracy of the SLR system while maintaining a low computational cost based on the human body pose in the form of 2D landmark locations. We first construct a skeleton graph based on 27 whole-body key points selected among 67 key points to address the high computational cost problem. Then, we utilize the multi-stream SL-GDN to extract features from the whole-body skeleton graph considering four streams. Finally, we concatenate the four different features and apply a classification module to refine the features and recognize corresponding sign classes. Our data-driven graph construction method increases the system’s flexibility and brings high generalizability, allowing it to adapt to varied data. We use two large-scale benchmark SLR data sets to evaluate the proposed model: The Turkish Sign Language data set (AUTSL) and Chinese Sign Language (CSL). The reported performance accuracy results demonstrate the outstanding ability of the proposed model, and we believe that it will be considered a great innovation in the SLR domain.

Korean Text to Gloss: Self-Supervised Learning approach

Natural Language Processing (NLP) has grown tremendously in recent years. Typically, bilingual, and multilingual translation models have been deployed widely in machine translation and gained vast attention from the research community. On the contrary, few studies have focused on translating between spoken and sign languages, especially non-English languages. Prior works on Sign Language Translation (SLT) have shown that a mid-level sign gloss representation enhances translation performance. Therefore, this study presents a new large-scale Korean sign language dataset, the Museum-Commentary Korean Sign Gloss (MCKSG) dataset, including 3828 pairs of Korean sentences and their corresponding sign glosses used in Museum-Commentary contexts. In addition, we propose a translation framework based on self-supervised learning, where the pretext task is a text-to-text from a Korean sentence to its back-translation versions, then the pre-trained network will be fine-tuned on the MCKSG dataset. Using self-supervised learning help to overcome the drawback of a shortage of sign language data. Through experimental results, our proposed model outperforms a baseline BERT model by 6.22%.

Chinese Sign Language Recognition Based on Two-stream CNN and LSTM Network

Sign language recognition is the use of computer technology to convert sign language into text or speech to facilitate the communication between deaf-mute people and normal people. This paper takes Chinese sign language words as the research object, and proposes a new method of sign language recognition based on two-stream 3D-CNN and LSTM network. First, the key frame extraction algorithm is used to remove redundant data frames in the original data, and then a two-stream3D-CNN is used to learn local hand change features and global trajectory features at the same time, and aggregated as the feature input of the video clip to the LSTM codec network. In order to focus on the video frames that express the meaning of sign language, a time attention mechanism is introduced in the LSTM encoding and decoding network. On the DEVISIGN-D sign language data set, an experiment was compared with three sign language recognition algorithms, the experimental results show that the method can identify Chinese isolated words sign language very well, with an accuracy rate of 98.4%.

Exploring Collection of Sign Language Videos through Crowdsourcing

Inadequate sign language data currently impedes advancement of sign language ML and AI. Training on existing datasets results in limited models due to small size, and lack of diverse signers in real-world settings. Complex labeling problems in particular often limit scale. In this work, we explore the potential for crowdsourcing to help overcome these barriers. To do this, we ran a user study with exploratory crowdsourcing tasks designed to support scalability: 1) to record videos of specific content -- thereby enabling automatic, scalable labeling -- and 2) to perform quality control checks for execution consistency -- further reducing post-processing requirements. We also provided workers with a searchable view of the crowdsourced dataset, to boost engagement and transparency and align with Deaf community values. Our user study included 29 participants using our exploratory tasks to record 1906 videos and perform 2331 quality control checks. Our results suggest that a crowd of signers may be able to generate high-quality recordings and perform reliable quality control, and that the signing community values visibility into the resulting dataset.

Deep transfer learning base on sequenced edge grid image technique for sign language recognition

Sign language is a visual-gestural language used by hearing impaired person, they modality the gesture to convey meaning. The main problem with sign language communication is ordinary people do not understand the sign language. Therefore, sign language is one of the challenging problems in machine learning. In this paper, researchers focus on visual-based methods and optimize the data preprocessing apply with existing sign language resources. Researchers propose an innovative technique for video processing called Sequenced Edge Grid Images (SEGI) for sign language recognition to interpret hand gesture, body movement, and facial expression. Researchers collected several of sign language data from the internet, the data including Thai sign language utilize in everyday life. The proposed technique was implemented with a convolutional neural network (CNN). The experiments showed SEGI with CNN has increases test accuracy rate with approximately 11% compared to static hand gesture images. Finally, researchers discovered a CNN structure suitable for dataset and examination data by transferring a pre-trained CNN. The fine-tuning with SEGI technique improved 99.8%, thus highest among all the methods. From the results data-preprocesses technique of dataset generation and deep transfer learning was an effective way to improve the accuracy of sign language recognition.

On the Reflexive KENDİ in Turkish Sign Language.

Linguistic analysis is improved when it includes language beyond the spoken modality. This paper uses sign language data to explore and advance cross-linguistic typologies of reflexives, constructions expressing that co-arguments of a predicate are also co-referent. In doing so, we also demonstrate that the lexical item KENDİ in Turkish Sign Language (henceforth, TİD) can function as a traditional reflexive, in addition to its previously documented emphatic functions. We further show that KENDİ is a DP-type reflexive, which helps to explain the emphatic usages of KENDİ that have been the focus of previous research. We end by outlining a plan for future research that can further probe and unify the superficially distinct functions of KENDİ and the typology of anaphoricity across modalities. Data for the present research comes from recently conducted fieldwork interviews with two signers of the İstanbul dialect of TİD, both of whom have been exposed to TİD since birth.

WearSign

Sign language translation (SLT) is considered as the core technology to break the communication barrier between the deaf and hearing people. However, most studies only focus on recognizing the sequence of sign gestures (sign language recognition (SLR)), ignoring the significant difference of linguistic structures between sign language and spoken language. In this paper, we approach SLT as a spatio-temporal machine translation task and propose a wearable-based system, WearSign, to enable direct translation from the sign-induced sensory signals into spoken texts. WearSign leverages a smartwatch and an armband of ElectroMyoGraphy (EMG) sensors to capture the sophisticated sign gestures. In the design of the translation network, considering the significant modality and linguistic gap between sensory signals and spoken language, we design a multi-task encoder-decoder framework which uses sign glosses (sign gesture labels) for intermediate supervision to guide the end-to-end training. In addition, due to the lack of sufficient training data, the performance of prior studies usually degrades drastically when it comes to sentences with complex structures or unseen in the training set. To tackle this, we borrow the idea of back-translation and leverage the much more available spoken language data to synthesize the paired sign language data. We include the synthetic pairs into the training process, which enables the network to learn better sequence-to-sequence mapping as well as generate more fluent spoken language sentences.We construct an American sign language (ASL) dataset consisting of 250 commonly used sentences gathered from 15 volunteers. WearSign achieves 4.7% and 8.6% word error rate (WER) in user-independent tests and unseen sentence tests respectively. We also implement a real-time version of WearSign which runs fully on the smartphone with a low latency and energy overhead.

From gestures to grammatical non-manuals in sign language: A case study of polar questions and negation in Hong Kong Sign Language

Previous studies propose that eyebrow raise in polar questions and headshakes in negations in sign languages arose from the incorporation of hearing speakers’ gestures in similar contexts into sign language grammar via grammaticalization. This paper reviews these proposals in light of Wilcox’s theory on gesture grammaticalization. According to Wilcox, there are two routes for gestures to enter sign language grammar: Route I, via lexicalization, and Route II, via prosody/intonation. Based on Cantonese and Hong Kong Sign Language (HKSL) data, we argue that eyebrow raise in polar questions in HKSL evolved from a paralinguistic gestural element into a grammatical marker via a language-internal process because eyebrow raise in question contexts is universal. Negative headshakes in Cantonese, though prevalent, are not incorporated into HKSL grammar. Instead, HKSL adopts backward head movements as optional negation markers. These findings support previous claims that headshakes are culture-specific. Both eyebrow raise in polar questions and backward head movements in negations in HKSL can be subsumed under Route II in Wilcox’s theory, along a language-internal evolutionary path. For sign languages that employ headshakes as negation markers, the headshakes are likely to have been borrowed from the hearing communities. This grammaticalization path fits neither Route I nor Route II of Wilcox’s theory because cross-modal borrowing is involved without an intermediate lexical stage.

From gesture- and sign-in-interaction to grammar: Fictive questions for relative clauses in signed languages

We discuss the use of the question-answer pattern for relativization across signed languages, with special attention for Catalan Sign Language. These are cases in which grammatical features of the interrogative construction used for genuine information-seeking questions also appear as the most unmarked, frequent, or only linguistic means of expressing restrictive and non-restrictive relative clauses, as well as appositives. This is intriguing, since relative clauses occur within one sentence and thus within one conversational turn, whereas the question-answer structure is prototypically dialogical, representing the turn-taking between addresser and addressee. We analyze such interrogative-like constructions as involving fictive interaction, the use of the conversation frame to structure cognition, discourse, and grammar (Pascual, 2006, 2014). We further suggest that the non-manual feature of eyebrow raising, which marks both information-seeking questions and relative clauses in Catalan Sign Language, became grammaticalized from a common non-obligatory gesture in the spoken Catalan of the surrounding hearing community. Hence, a gesture accompanying spoken language became a linguistic marker in a signed language, illustrating transfer between languages of different modalities. This is also presented as showing the emergence of grammar from situated intersubjective interaction (Li and Thompson, 1976; Sankoff and Brown, 1976). We make a case for the understanding of grammatical structure as primarily reflecting its mode of usage rather than some sui-generis Universal Grammar. This paper is based on the bibliographic study of 17 signed languages from different families and the qualitative analysis of own Catalan Sign Language data from different discourse genres.

Real-Time Block-Based Embedded CNN for Gesture Classification on an FPGA

This paper presents a block-based embedded convolutional neural network (CNN) for gesture classification on field-programmable gate array (FPGA) in real time. Gesture recognition is an important tool to spontaneous interact with human machine interface. Many CNN architectures using RGB images have been proposed for gesture classification. RGB based gesture classification may cause incorrect results under insufficient light or similar gestures. In addition, most of the CNN architectures cannot run in real time on edge devices due to their large number of parameters and DRAM data access. In this paper, a block-based CNN using RGB-D data is proposed for gesture classification. Adding depth images to RGB images boots the classification accuracy. A CNN architecture with block-based feature maps is built for embedded FPGA implementations. The total number of parameters of the proposed RGB-D embedded CNN (eCNN) model is only 0.17M and it achieves 99.96% and 99.88% accuracy with 32-bit floating point and 8-bit fixed point implementation for America Sign Language (ASL) data set. The RTL simulation of the proposed eCNN model has the average inference speed of 0.171 milliseconds at frequency of 250MHz for a single pair RGB-D image. Implemented on a FPGA integrated with Microsoft Kinect v2 achieve an inference time in 19.42 ms which achieves high accuracy and real-time performance.

Static hand gesture recognition of Arabic sign language by using deep CNNs

An Arabic sign language recognition using two concatenated deep convolution neural network models DenseNet121 & VGG16 is presented. The pre-trained models are fed with images, and then the system can automatically recognize the Arabic sign language. To evaluate the performance of concatenated two models in the Arabic sign language recognition, the red-green-blue (RGB) images for various static signs are collected in a dataset. The dataset comprises 220,000 images for 44 categories: 32 letters, 11 numbers (0:10), and 1 for none. For each of the static signs, there are 5000 images collected from different volunteers. The pre-trained models were used and trained on prepared Arabic sign language data. These models were used after some modification. Also, an attempt has been made to adopt two models from the previously trained models, where they are trained in parallel deep feature extractions. Then they are combined and prepared for the classification stage. The results demonstrate the comparison between the performance of the single model and multi-model. It appears that most of the multi-model is better in feature extraction and classification than the single models. And also show that when depending on the total number of incorrect recognize sign image in training, validation and testing dataset, the best convolutional neural networks (CNN) model in feature extraction and classification Arabic sign language is the DenseNet121 for a single model using and DenseNet121 & VGG16 for multi-model using.