Human Signs Research Articles

Mediated and non-mediated tactile fingerspelling: a comparative study

ABSTRACT In the development of communication devices for individuals who are Deafblind, a significant challenge is achieving a seamless transition from human-generated to technology-mediated communication. This study compares the intelligibility of the Australian Deafblind tactile fingerspelling alphabet rendered on the HaptiComm tactile communication device with the same alphabet articulated by a human signer. After a short training period, participants identified the 26 English alphabet letters in both the mediated (device) and non-mediated (human) conditions. Results indicated that while participants easily identified most letters in the non-mediated condition, the mediated condition was more difficult to decipher. Specifically, letters presented on the palm or near the index finger had significantly lower recognition rates. These findings highlight the need for further research on the tactile features of communication devices and emphasize the importance of refining these features to enhance the reliability and readability of mediated tactile communication produced through tactile fingerspelling.

Recognition of Wh-Question Sign Gestures in Video Streams using an Attention Driven C3D-BiLSTM Network

Human sign language is a visual and gestural means of communication used by people with hearing impairments to interact with others. It has the potential to enable interaction between individuals who struggle with verbal communication and those who lack the essential proficiency to comprehend sign language. Wh-questions constitute a substantial portion of daily sign language interactions. Automatic recognition of Wh-question words in sign language, encompassing interrogative words, has received limited attention in gesture recognition. In this context, a novel dataset named the American Question Sign Video dataset (AQSVd) has been introduced as a significant contribution. Recognizing Wh-question signs in video streams empowers individuals to effectively convey messages through hand movements and gestures, fostering inclusivity and accessibility in communication for deaf and hearing-impaired individuals. The paper proposes a Deep Convolutional-3D BiLSTM Multi-head Attention network for recognizing American Wh-Question word sign gestures in video streams. Incorporating Multi-head attention in the proposed model enhances its ability to capture intricate spatial and temporal features, essential for accurate gesture recognition. The proposed model is thoroughly evaluated across a range of datasets, such as INCLUDE50, WLASL-100, and our AQSVd dataset, achieving remarkable results. Specifically, our model demonstrated an outstanding 98.91% validation accuracy on the AQSVd dataset. The C3D-BiLSTM MHAttention model outperforms other state-of-the-art models, demonstrating its superiority in sign language recognition tasks. The proposed dataset and C3D-BiLSTM Multi-head Attention model contribute significantly to this field, offering potential benefits in education, human-robot interaction, and overall communication for the hearing-impaired community.

Machine Learning Based Customized Solution for Deafness and Mute People

Individuals with hearing or speech impairments often possess a higher degree of familiarity with sign language due to its widespread usage. Consequently, a noticeable communication divide exists between those with such impairments and the general populace. A pivotal solution in bridging this gap involves employing human sign language interpreters. Regrettably, the scarcity of sign language interpreters worldwide in comparison to the number of individuals with hearing or speech impairments has resulted in certain individuals being unable to bear the cost of a human interpreter each time they engage in conversation. To address this issue, it is imperative to automate communication in a manner that liberates the deaf community from dependence on human translators. This article centers on the development of an application capable of real-time conversion between American Sign Language and text, along with supplementary functionalities aimed at dismantling the communication barriers faced by individuals with hearing or speech challenges when interacting with the broader public. the primary objective of this application is to accurately perceive and interpret the user's sign language. The initial stage involves training the system to recognize and interpret signs utilizing algorithms for object recognition and motion tracking. For this purpose, a convolutional neural network model was trained employing a meticulously curated American Sign Language dataset. Subsequently, the recognized signs are translated into English, forming grammatically sound words. Incorporated within the system are a language translator and a virtual sign keyboard, augmenting its capabilities. Moreover, a text-to-text transformer, structured on an encoder-decoder framework, is harnessed to identify grammatical inaccuracies and generate coherent phrases. For a comprehensive understanding of the process, detailed elaboration will follow in subsequent sections.

Multi-Modal Multi-Channel American Sign Language Recognition

In this paper, we propose a machine learning-based multi-stream framework to recognize American Sign Language (ASL) manual signs and nonmanual gestures (face and head movements) in real time from RGB-D videos. Our approach is based on 3D Convolutional Neural Networks (3D CNNs) by fusing the multi-modal features including hand gestures, facial expressions, and body poses from multiple channels (RGB, Depth, Motion, and Skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3D CNN model. We collected a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera. Each video consists of 100 ASL manual signs, along with RGB channel, Depth maps, Skeleton joints, Face features, and HD face. The dataset is fully annotated for each semantic region (i.e. the time duration of each sign that the human signer performs). Our proposed method achieves 92.88% accuracy for recognizing 100 ASL sign glosses in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on a large-scale dataset, ChaLearn IsoGD, achieving the state-of-the-art results.

Exploring automatic text-to-sign translation in a healthcare setting

Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.

Bi-Directional American Sign Language and English Translator Using Machine Learning and Computer Vision

Abstract: American sign language is a language that is used by those individuals who are physically impaired and cannot communicate through common spoken languages. Due to lack of awareness and a small population of individuals using sign language, those who rely on it as their only means of communication require human sign language translators in order to be able to communicate with the more fortunate abled individuals. However, in reality, it is not alway practically feasible to have a human translator present at all times. That is why automation of a human sign language translator’s work can prove to be a more suitable alternative in a broader spectrum of use cases. A portable and practical sign language translator proves to be more socially and technically feasible than a human sign language translator. Keeping these set of circumstances in mind we are developing an application that allows two-way communication between a physically impaired individual and an abled individual. The application is designed for two categories of users : those who use ASL as their primary language for communication (disabled) and those who use English as their primary language for communication (abled). The primal aim is to curb the language barrier that exists between the 2 categories of individuals in the hopes of providing better opportunities in society for the less fortunate through the use of our application. We propose to use a machine learning, computer vision based model that is capable of learning and performing the translations between English text and American Sign Language and vice versa.

Application of virtual human sign language translation based on speech recognition

For the application problem of speech recognition to sign language translation, we conducted a study in two parts: improving speech recognition's effectiveness and promoting the application of sign language translation. The mainstream frequency-domain feature has achieved great success in speech recognition. However, it fails to capture the instantaneous gap in speech, and the time-domain feature makes up for this deficiency. In order to combine the advantages of frequency and time domain features, an acoustic architecture with a joint time domain encoder and frequency domain encoder is proposed. A new time-domain feature based on SSM (State-Space-Model) is proposed in the time- domain encoder and encoded using the GRU model. A new model, ConFLASH, is proposed in the frequency domain encoder, which is a lightweight model combining CNN and FLASH (a variant of the Transformer model). It not only reduces the computational complexity of the Transformer model but also effectively integrates the global modeling advantages of the Transformer model and the local modeling advantages of CNN. The Transducer structure is used to decode speech after the encoders are joined. This acoustic model is named GRU-ConFLASH- Transducer. On the self-built dataset and open-source dataset speechocean, it achieves optimal WER (Word Error Rate) of 2.6% and 4.7%. In addition, to better realize the visual application of sign language translation, a 3D virtual human model is designed and developed.

Sign Language Gesture to Speech Conversion Using Convolutional Neural Network

A genuine disability prevents a person from speaking. There are numerous ways for people with this condition to communicate with others, including sign language, which is one of the more widely used forms of communication. Human body language can be used to communicate with one another using sign language, where each word is represented by a specific sequence of gestures. The goal of the paper is to translate human sign language into speech that can interpret human gestures. Through a deep convolution neural network, we first construct the data-set, save the hand gestures in the database, and then use an appropriate model on these hand gesture visuals to test and train the system. When a user launches the application, it then detects the gestures that are saved inthe database and displays the corresponding results. By employing this system, it is possible to assist those who are hard of hearing while simultaneously making communication with them simpler for everyone else.

What about synthetic signing? A methodology for signer involvement in the development of avatar technology with generative capacity

Although signing avatar technology seems to be the only option currently available to serve sign language (SL) display in the context of applications which demand generative capacity from the part of the technology like in machine translation to SL, signing avatars have not yet been accepted by signers' communities. One major factor for this rejection is the feeling that technology is developed without the involvement of its actual users. Aiming to invite the signers' community into the process of signing avatar development, we have designed the shell methodological framework for signer-informed technology which is implemented as on-line surveys addressed to signer communities of different SLs. The surveys are communicated via focused on-line questionnaires with content of signing avatar performance that allows rating of various aspects of the produced SL synthetic signing by human signers. Here we report on the first survey application with content from the Greek Sign Language (GSL). The analysis of the obtained results is 2-fold: it highlights the significance of signer involvement and the provided feedback in the technological development of synthetic signing; in parallel it reveals those aspects of the survey setup that need fine-tuning before its next distribution cycles. The implementation of the first on-line survey can be found in: https://sign.ilsp.gr/slt-survey/.

Эволюция психики: аффект и разум

Introduction. Collision of affect and intellect determines the course of human psychic evolution and sets a system of coordinates for personality development. Throughout life, every person has to reconcile his or her desires with the requirements of reason. Objective: to study the development of the psyche from the perspective of the relationship between affective and intellectual forms of object-oriented activity. Methodology and methods: the principle of activity and the cultural-historical approach. Results of the research. The study starts with finding out the role of affects in the communication processes, where the psyche deals with another psyche: they are exchanging affective signals, the affective waves collide and form associations, etc. In the simplest case, the affective and expressive reaction of a living body A serves as a signal for action of a body B. The regular exchange of affects coordinates the activity of bodies A and B in such a way that they form a single organism or community. The highest, universal psychic form of communication is speech, which allows us to express any affect and to form complex communities with the help of stereotypical signals. Natural speech is an emotional “contagion” (Vygotsky) that ensures coordination between community members. Human sign language, including the verbal one, emerges from the fusion of thought and language processes – the synthesis of affects with concepts. Vygotsky established that in the behaviour of animals, and first in children too, thinking and speech “completely destroy” each other, while in cultural person, thinking becomes inseparable from speech. In the article, this transformation is explained based on understanding of thinking and speech as the contrary forms of object-oriented activity. Affective speech turns into rational language as the individual assimilates ideas and meanings embodied in cultural artefacts. This assimilation takes place in the course of jointly-divided activity with the artefacts. Concept is an internalised idea of a thing. Vygotsky rightly saw concepts as “the key to all genuinely human” and the source of our freedom, interpreted as “the affect in the concept”. Following Spinoza and Vygotsky, the author considers the affect and the concept as two coordinate axes of the human psyche, and cultural-historical psychology should study the concrete forms of synthesizing the affect with the concept in cultural history and in the process of forming the child’s psyche.

Attitudes Toward Signing Avatars Vary Depending on Hearing Status, Age of Signed Language Acquisition, and Avatar Type.

The use of virtual humans (i.e., avatars) holds the potential for interactive, automated interaction in domains such as remote communication, customer service, or public announcements. For signed language users, signing avatars could potentially provide accessible content by sharing information in the signer's preferred or native language. As the development of signing avatars has gained traction in recent years, researchers have come up with many different methods of creating signing avatars. The resulting avatars vary widely in their appearance, the naturalness of their movements, and facial expressions—all of which may potentially impact users' acceptance of the avatars. We designed a study to test the effects of these intrinsic properties of different signing avatars while also examining the extent to which people's own language experiences change their responses to signing avatars. We created video stimuli showing individual signs produced by (1) a live human signer (Human), (2) an avatar made using computer-synthesized animation (CS Avatar), and (3) an avatar made using high-fidelity motion capture (Mocap avatar). We surveyed 191 American Sign Language users, including Deaf (N = 83), Hard-of-Hearing (N = 34), and Hearing (N = 67) groups. Participants rated the three signers on multiple dimensions, which were then combined to form ratings of Attitudes, Impressions, Comprehension, and Naturalness. Analyses demonstrated that the Mocap avatar was rated significantly more positively than the CS avatar on all primary variables. Correlations revealed that signers who acquire sign language later in life are more accepting of and likely to have positive impressions of signing avatars. Finally, those who learned ASL earlier were more likely to give lower, more negative ratings to the CS avatar, but we did not see this association for the Mocap avatar or the Human signer. Together, these findings suggest that movement quality and appearance significantly impact users' ratings of signing avatars and show that signed language users with earlier age of ASL acquisition are the most sensitive to movement quality issues seen in computer-generated avatars. We suggest that future efforts to develop signing avatars consider retaining the fluid movement qualities integral to signed languages.

Pakistan sign language translation tool in educational setting: teachers perspective

PurposeThe current study aims to determine the viability of the tool developed by Abbas and Sarfraz (2018) to translate English speech and text to Pakistan Sign Language (PSL) with bilingual subtitles.Design/methodology/approachFocus group interviews of 30 teachers of a Pakistani private university were conducted; who used the PSL translation tool in their classrooms for lecture delivery and communication with the deaf students.FindingsThe findings of the study determined the viability of the developed tool and showed that it is helpful in teaching deaf students efficiently. With the availability of this tool, teachers are not dependent on human sign language (SL) interpreters in their classrooms.Originality/valueOverall, this tool is an effective addition to educational technology for special education. Due to the lack of Sign Language (SL) understanding, learning resources and availability of human SL interpreters in Pakistan, institutions feel dependency and scarcity to educate deaf students in a classroom. Unimpaired people and especially teachers face problems communicating with deaf people to arrange one interpreter for a student(s) in multiple classes at the same time which creates a communication gap between a teacher and a deaf student.

Design and Development of IoT Device that Recognizes Hand Gestures using Sensors

A hand gesture is a non-verbal means of communication involving the motion of fingers to convey information. Hand gestures are used in sign language and are a way of communication for deaf and mute people and also implemented to control devices too. The purpose of gesture recognition in devices has always been providing the gap between the physical world and the digital world. The way humans interact among themselves with the digital world could be implemented via gestures using algorithms. Gestures can be tracked using gyroscope, accelerometers, and more as well. So, in this project we aim to provide an electronic method for hand gesture recognition that is cost-effective, this system makes use of flex sensors, ESP32 board. A flex sensor works on the principle of change in the internal resistance to detect the angle made by the user’s finger at any given time. The flexes made by hand in different combinations amount to a gesture and this gesture can be converted into signals or as a text display on the screen. A smart glove is designed which is equipped with custom-made flex sensors that detect the gestures and convert them to text and an ESP32 board, the component used to supplement the gestures detected by a flex sensor. This helps in identifying machines the human sign language and perform the task or identify a word through hand gestures and respond according to it.

Motionlets Matching With Adaptive Kernels for 3-D Indian Sign Language Recognition

Recognizing human gestures in sign language is a complex and challenging task. Human sign language gestures are a combination of independent hand and finger articulations, which are sometimes performed in coordination with the head, face, and body. 3-D motion capture of sign language involves recording 3-D sign videos that are often affected by interobject or self-occlusions, lighting, and background. This paper proposes characterization of sign language gestures articulated at different body parts as 3-D motionlets, which describe the signs with a subset of joint motions. A two-phase fast algorithm identifies 3-D query signs from an adaptively ranked database of 3-D sign language. Phase-I process clusters all human joints into motion joints (MJ) and nonmotion joints (NMJ). The relation between MJ and NMJ is analyzed to categorically segment the database into four motionlet classes. Phase-II process investigates the relation within the motion joints to represent shape information of a sign as 3-D motionlets. The 4-class sign database features three adaptive motionlet kernels. A simple kernel matching algorithm is used to rank the database according to the highest-ranked query sign. The proposed method is sign invariant to temporal misalignment and can characterize sign language based on a 3-D spatiotemporal framework. In this paper, five 500-word Indian sign language data sets were used to evaluate the proposed model. The experimental results reveal that the method proposed here improved recognition compared with the state-of-the-art 3-D action recognition methods.

Study of sign segmentation in the text of Chinese sign language

The natural language processing (NLP) of sign language aims to make human sign language "understandable" to computers. In achieving this goal, the text of sign language should first be segmented in...

Study of sign segmentation in the text of Chinese sign language

The natural language processing (NLP) of sign language aims to make human sign language "understandable" to computers. In achieving this goal, the text of sign language should first be segmented into sign sequences for computers to recognize. This segmentation process constitutes the basis for the information processing of sign language. With an aim to solve the problems in expressing Chinese sign language (CSL), this paper analyzes the lexical features of CSL and discusses various sign segmentation algorithms used in obtaining computer-read files. Sign segmentation involves two main approaches: The first is rule based, whereas the second is statistics based. Backward maximum matching (BMM) is an important rule-based method widely used in Chinese NLP fields. The recently proposed conditional random fields (CRFs) have also demonstrated excellent performance as a statistical method in international tests. In this study, both the BMM and CRFs methods are employed on the same dataset to explore the practical issues in the sign segmentation of CSL. The results of the CRFs method are then presented and discussed. Our corpus contains only hundreds of sentences; therefore, cross-validation based on CRFs is also performed to avoid the unreliable function that may arise from using an exceedingly small corpus scale within limited processing time. Specifically, three-group twofold cross-validation is applied to analyze the design of the annotation specification and the selection of a feature template. The results validate the effectiveness of our proposed segmentation strategy and confirm that CRFs outperform the BMM method. The proposed approach yields an F-score of 77.4% in sign segmentation in the CSL corpus. The CRFs perform effectively in sign segmentation because they can capture the arbitrary, overlapping features of the input in a Markov model. However, to obtain more satisfactory results, we must rely on the technological development of the sign language corpus.

Conversation of Sign Language to Speech with Human Gestures

Abstract Inability to speak is considered to be true disability. People with this disability use different modes to communicate with others, there are n number of methods available for their communication one such common method of communication is sign language. Sign language allows people to communicate with human body language; each word has a set of human actions representing a particular expression. The motive of the paper is to convert the human sign language to Voice with human gesture understanding and motion capture. This is achieved with the help of Microsoft Kinect a motion capture device from Microsoft. There are a few systems available for sign language to speech conversion but none of them provide natural user interface. For consideration if a person who has a disability to speak can stand perform the system and the system converts the human gestures as speech and plays it loud so that the person could actually communicate to a mass crowd gathering. Also the system is planned in bringing high efficiency for the users for improved communication.

Effect of Displaying Human Videos During an Evaluation Study of American Sign Language Animation

Many researchers internationally are studying how to synthesize computer animations of sign language; such animations have accessibility benefits for people who are deaf and have lower literacy in written languages. The field has not yet formed a consensus as to how to best conduct evaluations of the quality of sign language animations, and this article explores an important methodological issue for researchers conducting experimental studies with participants who are deaf. Traditionally, when evaluating an animation, some lower and upper baselines are shown for comparison during the study. For the upper baseline, some researchers use carefully produced animations, and others use videos of human signers. Specifically, this article investigates, in studies where signers view animations of sign language and are asked subjective and comprehension questions, whether participants differ in their subjective and comprehension responses when actual videos of human signers are shown during the study. Through three sets of experiments, we characterize how the Likert-scale subjective judgments of participants about sign language animations are negatively affected when they are also shown videos of human signers for comparison -- especially when displayed side-by-side. We also identify a small positive effect on the comprehension of sign language animations when studies also contain videos of human signers. Our results enable direct comparison of previously published evaluations of sign language animations that used different types of upper baselines -- video or animation. Our results also provide methodological guidance for researchers who are designing evaluation studies of sign language animation or designing experimental stimuli or questions for participants who are deaf.

Effect of spatial reference and verb inflection on the usability of sign language animations

Computer-generated animations of American Sign Language (ASL) can improve the accessibility of information, communication, and services for the significant number of deaf adults in the US with difficulty in reading English text. Unfortunately, there are several linguistic aspects of ASL that current automatic generation or translation systems cannot produce (or are time-consuming for human animators to create). To determine how important such phenomena are to user satisfaction and the comprehension of ASL animations, studies were conducted in which native ASL signers evaluated ASL animations with and without: establishment of spatial reference points around the virtual human signer representing entities under discussion, pointing pronoun signs, contrastive role shift, and spatial inflection of ASL verbs. It was found that adding these phenomena to ASL animations led to a significant improvement in user comprehension of the animations, thereby motivating future research on automating the generation of these animations.

Human-inspired search for redundancy in automatic sign language recognition

Human perception of sign language can serve as inspiration for the improvement of automatic recognition systems. Experiments with human signers show that sign language signs contain redundancy over time. In this article, experiments are conducted to investigate whether comparable redundancies also exist for an automatic sign language recognition system. Such redundancies could be exploited, for example, by reserving more processing resources for the more informative phases of a sign, or by discarding uninformative phases. In the experiments, an automatic system is trained and tested on isolated fragments of sign language signs. The stimuli used were similar to those of the human signer experiments, allowing us to compare the results. The experiments show that redundancy over time exists for the automatic recognizer. The central phase of a sign is the most informative phase, and the first half of a sign is sufficient to achieve a recognition performance similar to that of the entire sign. These findings concur with the results of the human signer studies. However, there are differences as well, most notably the fact that human signers score better on the early phases of a sign than the automatic system. The results can be used to improve the automatic recognizer, by using only the most informative phases of a sign as input.