Sign Language Synthesis Research Articles

SPEECH TO SIGN CONVERSION USING NATURAL LANGUAGE PROCESSING

Speech to sign conversion is one essential piece of technology that helps the hearing community and the hearing impaired communicate. This paper presents a novel approach utilizing Natural Language Processing (NLP) techniques for speech to sign conversion. The system that is being suggested leverages advanced NLP algorithms to transcribe spoken language into text, which is thereafter converted to sign language. The process involves several key steps: speech recognition and sign language synthesis. The synthesized sign language gestures are generated based on a comprehensive database of sign language vocabulary and grammar rules. The proposed speech to sign conversion system demonstrates promising outcomes concerning accuracy, efficiency, and usability. By harnessing the power of NLP, this technology possesses the capacity to significantly enhance communication accessibility for the hearing-impaired community, facilitating seamless interaction n a number of fields, such as education and healthcare and social communication. Key Words: Speech Recognition, Natural Language Processing, Tokenization, Lemmatization, Parsing.

Research on the improved gesture tracking algorithm in sign language synthesis

Research on the improved gesture tracking algorithm in sign language synthesis

Natural synthesis of productive forms from structured descriptions of sign language

Natural animation of sign language directly from linguistic descriptions continues to be a challenge especially in cases where the forms involved are more productive, such as geometric depictions. Prior work laid the foundation for natural sign language synthesis with the Paula animation system directly from AZee linguistic descriptions. This paper considers more elaborate discourse, composed of several clauses linked together by the overall meaning and involving largely productive signing. We make the case that one of the keys to natural animation of such discourse lies also in the segments between the typically annotated signs, in other words on the segments traditionally termed “transitions”. By studying an example discourse video and the corresponding motion capture, we progressively build an efficient linguistic description of it and specify how to animate it naturally.

A survey on the animation of signing avatars: From sign representation to utterance synthesis

Signing avatars make it possible for deaf people to access information in their preferred language. However, sign language synthesis represents a challenge for the computer animation community as the motions generated must be realistic and have a precise semantic meaning. In this article, we distinguish the synthesis of isolated signs deprived of any contextual inflections from the generation of full sign language utterances. In both cases, the animation engine takes as input a representation of the synthesis objective to create the final animation. Because of their spatiotemporal characteristics, signs and utterances cannot be described by a sequential representation like phonetics in spoken languages. For this reason, linguistic and gestural studies have aimed to capture the typical and special features of signs and sign language syntax to promote different sign language representations. Those sign representations can then be used to produce an avatar animation thanks to sign synthesis techniques based on keyframes, procedural means or data-driven approaches. Novel utterances can also be generated using concatenative or articulatory techniques.This article constitutes a survey of (i) the challenges specific to sign languages avatars, (ii) the sign representations developed in order to synthesize isolated signs, (iii) the possible sign synthesis approaches, (iv) the different utterance specifications, and (v) the challenges and animation techniques for generating sign language utterances.

Learning Three-dimensional Skeleton Data from Sign Language Video

Data for sign language research is often difficult and costly to acquire. We therefore present a novel pipeline able to generate motion three-dimensional (3D) skeleton data from single-camera sign language videos only. First, three recurrent neural networks are learned to infer the three-dimensional position data of body, face, and finger joints for a high resolution of the signer’s skeleton. Subsequently, the angular displacements of all joints over time are estimated using inverse kinematics and mapped to a virtual sign avatar for animation. Last, the generated data are evaluated in detail, including a sign language recognition and sign language synthesis scenario. Utilizing a neural word classifier trained on real motion capture data, we reliably classify word segments built from our newly generated position data with similar accuracy as motion capture data (absolute difference 3.8%). Furthermore, qualitative evaluation of sign animations shows that the avatar performs natural movements that are comprehensible and resemble animations created with original motion capture data.

Full Skeleton-based Virtual Human Animation: An Improved Design for SL Avatars

ABSTRACTThis paper presents a novel skeleton design for avatar animation. This design includes new bones, whose purpose is to meet the requirements of sign language (SL) synthesis. SL synthesis presents particular challenges regarding the processing of phonetic descriptions of signs to create animations. Although traditional animation techniques have solved the synthesis of signs, the proposed skeleton design allows for the simplification of the phonetic processing, the inverse kinematics calculations, and the collision avoidance processes. The proposed design has been tested to demonstrate that the generated animations have the same quality as the ones produced with state-of-the-art techniques. Importantly, the proposed structure reduces the time required for generating an animation by 52% compared to existing solutions and requires less storage resources.

Capture Surface Light Field for Gesture with Sparse Multi-view Videos

A Surface Light Field (SLF) is a function that assigns a color to each ray originating on a surface, it reveals light information surrounding a scene or object. This paper presents a framework for capturing SLF for gestures with sparse multi-view videos. Captured SLF is saved in discrete SLF matrix sequence, and it can be used for rendering images for new view point and moment. In order to process dynamics of gesture light field, a prior geometry model and related shape deforming are introduced into the frame, which relaxes requirement for accuracy of the prior shape model and make the light field available for scenes with complex self-occlusion such as sign language words. Also because the dynamic SLF sequence has same geometry topological structure, it is convenient for interpolation such as what is used in sign language synthesis.

Lexicon Formalization for a Computer System of Sign Language Synthesis

In this paper, we give an analytical survey of peculiarities of the Russian sign language and calc signed (gesture) speech, including state-of-the-art of sign lexicons and grammatical constructions, as well as methods for formalization of sign lexicon items. In the course of the multidisciplinary research, a virtual 3D model of human being’s avatar has been adopted for the Russian sign language synthesis, and a multipurpose model of multimodal audio-visual synthesizer aimed for text to Russian auditory speech and calc signed speech transformation.

Research of Sign Language Synthesis Based on VRML

Sign language is to help the deaf and normal hearing people natural communication and computer assisted instruction. Through the analysis of language features, and proposed one kind based on the VRML human body modeling and virtual human based on context of gesture smoothing algorithm, thus the sign language synthesis research and implementation.

Chinese Sign Language animation generation considering context

Sign language (SL) is a kind of natural language for the deaf. Chinese Sign Language (CSL) synthesis aims to translate text into virtual human animation, which makes information and service accessible to the deaf. Generally, sign language animation based on key frames is realized by concatenating sign words captured independently. That means a sign language word has the same pattern in diverse context, which is different from realistic sign language expression. This paper studies the effect of context on manual gesture and non-manual gesture, and presents a method for generating stylized manual gesture and non-manual gesture according to the context. Experimental results show that synthesized sign language animation considering context based on the proposed method is more accurate and intelligible than that irrespective of context.

Spanish Sign Language synthesis system

This work presents a new approach to the synthesis of Spanish Sign Language (LSE). Its main contributions are the use of a centralized relational database for storing sign descriptions, the proposal of a new input notation and a new avatar design, the skeleton structure of which improves the synthesis process.The relational database facilitates a highly detailed phonologic description of the signs that include parameter synchronization and timing. The centralized database approach has been introduced to allow the representation of each sign to be validated by the LSE National Institution, FCNSE.The input notation, designated HLSML, presents multiple levels of abstraction compared with current input notations. Redesigned input notation is used to simplify the description and the manual definition of LSE messages.Synthetic messages obtained using our approach have been evaluated by deaf users; in this evaluation a maximum recognition rate of 98.5% was obtained for isolated signs and a recognition rate of 95% was achieved for signed sentences.

Chinese Sign Language Synthesis System on Mobile Device

In this paper, we proposed an effective approach for Chinese Sign Language (CLS) Synthesis System on mobile device. It can translate messages or other text information into Chinese sign languages in real time. In this system, it mainly includes two parts: construction of motion database and sign language synthesis based on text-driven. Construction of motion database is a prerequisite for the Chinese sign language synthesis system. Its motion quality, motion data structure and motion data size together determine the final performance of system. We also introduce a method for motion smoothing in synthesis module. We have integrated two parts on a mobile device, which can render a series of sign language motions according to specific input text from Mobile Internet.

Hybrid paradigm for Spanish Sign Language synthesis

This work presents a hybrid approach to sign language synthesis. This approach allows the hand-tuning of the phonetic description of the signs, which focuses on the time aspect of the sign. Therefore, the approach retains the capacity for the performing of morpho-phonological operations, like notation-based approaches, and improves the synthetic signing performance, such as the hand-tuned animations approach. The proposed approach simplifies the input message description using a new high-level notation and storage of sign phonetic descriptions in a relational database. Such relational database allows for more flexible sign phonetic descriptions; it also allows for a description of sign timing and the synchronization between sign phonemes. The new notation, named HLSML, is a gloss-based notation focusing on message description in it. HLSML introduces several tags that allow for the modification of the signs in the message that defines dialect and mood variations, both of which are defined in the relational database, and message timing, including transition durations and pauses. A new avatar design is also proposed that simplifies the development of the synthesizer and avoids any interference with the independence of the sign language phonemes during animation. The obtained results showed an increase of the sign recognition rate compared to other approaches. This improvement was based on the active role that the sign language experts had in the description of signs, which was the result of the flexibility of the sign storage approach. The approach will simplify the description of synthesizable signed messages, thus facilitating the creation of multimedia-signed contents.

Interactive sign language synthesis based on adaptive display resolution visibility for ubiquitous devices

AbstractThis paper presents a novel framework that synthesize Chinese sign language (CSL) animation on ubiquitous platform‐independent devices with consideration of the self‐adaptive rendering capability of the device display screen. The display resolution visibility (DRV) factor is introduced here as the measurement of the system's adaptability. Both the 3D virtual human model and sign animation scripts are simplified according to the DRV Factor of the ubiquitous device. Consequently, a real‐time transmission scheme is proposed for sign language animation over a heterogeneous network. Simulations results show that these innovations significantly reduce the transmission bandwidth and work very well on the ubiquitous devices. Copyright © 2011 John Wiley & Sons, Ltd.

A prototype Greek text to Greek Sign Language conversion system

In this article, a prototype Greek text to Greek Sign Language (GSL) conversion system is presented. The system is integrated into an educational platform that addresses the needs of teaching GSL grammar and was developed within the SYNENNOESE project (Efthimiou et al. 2004a. Developing an e-learning platform for the Greek sign language. In: K. Miesenberger, J. Klaus, and W. Zagler, eds. Computer helping people with special needs, Lecture Notes in Computer Science, Springer-Verlag, vol. 3118, 1107–1113). The detailed implementation of the language-processing component of a Greek text to GSL conversion system is provided, focusing upon the inherent problems of knowledge elicitation of sign language (SL) grammar and its implementation within a parser framework. It is based on an SL dictionary (Efthimiou et al. 2004a) database of coded GSL knowledge. The proposed system has been designed and implemented after considering most state-of-the-art SL machine translation or Conversion systems, such as Vsigns (Papadogiorgaki et al. 2004. VSigns – a virtual sign synthesis web tool. In: Proceedings of Sixth COST 276 Workshop on Information and Knowledge Management for Integrated Media Communication, May 2004, Thessaloniki, Greece), ZARDOZ (Veale et al. 1998. The challenges of cross-modal translation: English to sign language translation in the ZARDOZ system. Machine Translation, 13, 81–106) and SignSynth (Angus 2001. SignSynth: a sign language synthesis application using Web3D and Perl. In: Gesture and Sign Lanauage in Human–Computer Interaction. London, UK: International Gesture Workshop), and taking into account their advantages and disadvantages. The overall architecture is innovative since other existing systems either do not consider the GSL or they cannot be effectively applied on sentences but just on single words. The system is demonstrable on any conventional PC.

Sign language synthesis

In this paper, I describe the summary of what I plan to do for my PhD thesis on sign language synthesis. The modelling of the skeleton of virtual characters would lead to improvements in the quality of animations. Thus the first step is to study how to build such a model, by analysing corpora data and integrating biological data. Then, we will implement the model and confront it to users to evaluate it.

CSLML: a markup language for expressive Chinese sign language synthesis

This paper presents a Chinese Sign Language Markup Language (CSLML), which is developed for expressive Chinese sign language synthesis by introducing features and structure of sign language prosody...

CSLML: a markup language for expressive Chinese sign language synthesis

AbstractThis paper presents a Chinese Sign Language Markup Language (CSLML), which is developed for expressive Chinese sign language synthesis by introducing features and structure of sign language prosody. The tags of CSLML are divided into two levels: function level and phonetic level. Function level provides abstract information about signed content and prosody, so it facilitates text annotating for text‐driven automatic synthetic system and adapts to diversified synthetic methods, such as motion capture animation or image‐based synthesis, which may not be good at processing lower‐level information. Phonetic level provides detailed behavioral manners based on phonetics and phonology to interpret the meaning in function level. It facilitates creation and edit of any motions. The two levels co‐exist in CSLML documents and high‐level description can be mapped into corresponding low‐level behavior to provide one‐to‐many variability and expression of synthesis. Therefore, we also introduce a framework for this mapping processing and exhibit results of our animated prototype system based on this framework. Copyright © 2009 John Wiley & Sons, Ltd.

Application of a Sign Language Synthesis System in Digital Library Services

Information Accessibility for disabled people is one of the most important design criteria for the China National Digital Library (CNDL) development. Sign language synthesis systems are effective to provide information services for people with hearing and speech impairments. This paper presents a framework of a sign language synthesis system application in CNDL, as well as discusses the relevant technologies applied in the system. CNDL has been a real practice area for the sign language synthesis research.

Linguistic modelling and language-processing technologies for Avatar-based sign language presentation

Sign languages are the native languages for many pre-lingually deaf people and must be treated as genuine natural languages worthy of academic study in their own right. For such pre-lingually deaf, whose familiarity with their local spoken language is that of a second language learner, written text is much less useful than is commonly thought. This paper presents research into sign language generation from English text at the University of East Anglia that has involved sign language grammar development to support synthesis and visual realisation of sign language by a virtual human avatar. One strand of research in the ViSiCAST and eSIGN projects has concentrated on the generation in real time of sign language performance by a virtual human (avatar) given a phonetic-level description of the required sign sequence. A second strand has explored generation of such a phonetic description from English text. The utility of the conducted research is illustrated in the context of sign language synthesis by a preliminary consideration of plurality and placement within a grammar for British Sign Language (BSL). Finally, ways in which the animation generation subsystem has been used to develop signed content on public sector Web sites are also illustrated.