Gesture Animation Research Articles

Zero-shot style transfer for gesture animation driven by text and speech using adversarial disentanglement of multimodal style encoding.

Modeling virtual agents with behavior style is one factor for personalizing human-agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero-shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive; while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of a speaker whose data are not part of the training phase, without requiring any further training or fine-tuning. The first goal of our model is to generate the gestures of a source speaker based on the content of two input modalities-Mel spectrogram and text semantics. The second goal is to condition the source speaker's predicted gestures on the multimodal behavior style embedding of a target speaker. The third goal is to allow zero-shot style transfer of speakers unseen during training without re-training the model. Our system consists of two main components: (1) a speaker style encoder network that learns to generate a fixed-dimensional speaker embedding style from a target speaker multimodal data (mel-spectrogram, pose, and text) and (2) a sequence-to-sequence synthesis network that synthesizes gestures based on the content of the input modalities-text and mel-spectrogram-of a source speaker and conditioned on the speaker style embedding. We evaluate that our model is able to synthesize gestures of a source speaker given the two input modalities and transfer the knowledge of target speaker style variability learned by the speaker style encoder to the gesture generation task in a zero-shot setup, indicating that the model has learned a high-quality speaker representation. We conduct objective and subjective evaluations to validate our approach and compare it with baselines.

Processing Real-Life Recordings of Facial Expressions of Polish Sign Language Using Action Units

Automatic translation between the national language and sign language is a complex process similar to translation between two different foreign languages. A very important aspect is the precision of not only manual gestures but also facial expressions, which are extremely important in the overall context of a sentence. In this article, we present the problem of including facial expressions in the automation of Polish-to-Polish Sign Language (PJM) translation-this is part of an ongoing project related to a comprehensive solution allowing for the animation of manual gestures, body movements and facial expressions. Our approach explores the possibility of using action unit (AU) recognition in the automatic annotation of recordings, which in the subsequent steps will be used to train machine learning models. This paper aims to evaluate entropy in real-life translation recordings and analyze the data associated with the detected action units. Our approach has been subjected to evaluation by experts related to Polish Sign Language, and the results obtained allow for the development of further work related to automatic translation into Polish Sign Language.

БАСТАУЫШ МЕКТЕПТЕ ПӘНАРАЛЫҚ БАЙЛАНЫС АРҚЫЛЫ ЖАҢАРТЫЛҒАН БАҒДАРЛАМАМЕН ОҚУШЫЛАРДЫҢ ТАНЫМДЫҚ ҚАБІЛЕТТЕРІН ҚАЛЫПТАСТЫРУ

In this article shown the effectiveness of introducing PBL and CLIL learning technologies into the educational process in accordance with modern learning conditions and the possibility of increasing productivity, as well as interest among students. In the course of the research, special attention was paid to the application of innovative methods in groups with bilingual education, which requires new approaches both in teaching and in the criteria for assessing achievements. The new technologies obtained in the course of practical application in the framework of teaching topical issues in the field of biology and ecology have led to opportunities both to expand the abilities of students in narrow areas of science and to allow directing education in an independent direction. In addition, the article developed new criteria for assessing learning outcomes in PBL on given topics, which made it possible to assess the achievements of students in the framework of a nontraditional innovative method and which are advisory in nature for bilingual education in the field of biology, biotechnology and ecology. As a PBL product in the experimental groups, the search topic “Drinking water resources in the southern region of Kazakhstan” was proposed, which is of current practical importance. In addition, CLIL technology teaching methods were proposed to determine the level of independent learning: gesture animation, the use of visual aids and props, introductory scaffolding: research and animation.

NEW APPROACHES TO LEARNING USING PBL AND CLIL TECHNOLOGIES AS WAYS TO INCREASE THE PRODUCTIVITY OF BILINGUAL TEACHING OF BIOLOGY

In this article shown the effectiveness of introducing PBL and CLIL learning technologies into the educational process in accordance with modern learning conditions and the possibility of increasing productivity, as well as interest among students. In the course of the research, special attention was paid to the application of innovative methods in groups with bilingual education, which requires new approaches both in teaching and in the criteria for assessing achievements. The new technologies obtained in the course of practical application in the framework of teaching topical issues in the field of biology and ecology have led to opportunities both to expand the abilities of students in narrow areas of science and to allow directing education in an independent direction. In addition, the article developed new criteria for assessing learning outcomes in PBL on given topics, which made it possible to assess the achievements of students in the framework of a nontraditional innovative method and which are advisory in nature for bilingual education in the field of biology, biotechnology and ecology. As a PBL product in the experimental groups, the search topic “Drinking water resources in the southern region of Kazakhstan” was proposed, which is of current practical importance. In addition, CLIL technology teaching methods were proposed to determine the level of independent learning: gesture animation, the use of visual aids and props, introductory scaffolding: research and animation.

Создание анимированного персонажа для системы компьютерного сурдоперевода

In recent years, much attention has been paid to people with disabilities around the world. Creating a barrier-free environment and the necessary conditions for their training and communication is one of the important tasks of any modern state. Since everyone should have equal opportunities to communicate with the outside world. In our country, it is also important and necessary to create a system of automatic sign language translation from Kazakh to sign language for people with hearing disabilities. This article reveals the prospects of using an animated character (avatar) to demonstrate gestures, the features of creating an animated character and applying sign language notation to control it. The avatar is aimed at showing the results of translating the Kazakh text into sign language based on the previously created dictionary of notation of gestures of the Kazakh sign language (KSL). When developing an automatic sign language translation system, the meaning of the translated text is revealed, it is translated into the grammar of the KSL. The compiled dictionary is then used to search for words in the dictionary, present them as notation, and display them using an avatar. To create an avatar model, software tools such as MakeHuman, Photoshop, and Blender are used in stages. Gesture animations are set up in the modern cross-platform graphics package Unity3D using libraries of "clips" created based on the dictionary of notations of gestures of the KSL.

Affective synthesis and animation of arm gestures from speech prosody

In human-to-human communication, speech signals carry rich emotional cues that are further emphasized by affect-expressive gestures. In this regard, automatic synthesis and animation of gestures accompanying affective verbal communication can help to create more naturalistic virtual agents in human-computer interaction systems. Speech-driven gesture synthesis can map emotional cues of the speech signal to affect-expressive gestures by modeling complex variability and timing relationships of speech and gesture. In this paper, we investigate the use of continuous affect attributes, which are activation, valence and dominance, for speech-driven affective synthesis and animation of arm gestures. To this effect, we present a statistical framework based on hidden semi-Markov models (HSMM), where states are gestures and observations are speech-prosody and continuous affect attributes. The proposed framework is evaluated considering four distinct HSMM structures which differ by their emission distributions. Evaluations are performed over the USC CreativeIT database in a speaker-independent setup. Among the four statistical structures, the conditional structure, which models observation distributions as prosody given affect, achieves the best performance under both objective and subjective evaluations.

Multimodal analysis of speech and arm motion for prosody-driven synthesis of beat gestures

We propose a framework for joint analysis of speech prosody and arm motion towards automatic synthesis and realistic animation of beat gestures from speech prosody and rhythm. In the analysis stage, we first segment motion capture data and speech audio into gesture phrases and prosodic units via temporal clustering, and assign a class label to each resulting gesture phrase and prosodic unit. We then train a discrete hidden semi-Markov model (HSMM) over the segmented data, where gesture labels are hidden states with duration statistics and frame-level prosody labels are observations. The HSMM structure allows us to effectively map sequences of shorter duration prosodic units to longer duration gesture phrases. In the analysis stage, we also construct a gesture pool consisting of gesture phrases segmented from the available dataset, where each gesture phrase is associated with a class label and speech rhythm representation. In the synthesis stage, we use a modified Viterbi algorithm with a duration model, that decodes the optimal gesture label sequence with duration information over the HSMM, given a sequence of prosody labels. In the animation stage, the synthesized gesture label sequence with duration and speech rhythm information is mapped into a motion sequence by using a multiple objective unit selection algorithm. Our framework is tested using two multimodal datasets in speaker-dependent and independent settings. The resulting motion sequence when accompanied with the speech input yields natural-looking and plausible animations. We use objective evaluations to set parameters of the proposed prosody-driven gesture animation system, and subjective evaluations to assess quality of the resulting animations. The conducted subjective evaluations show that the difference between the proposed HSMM based synthesis and the motion capture synthesis is not statistically significant. Furthermore, the proposed HSMM based synthesis is evaluated significantly better than a baseline synthesis which animates random gestures based on only joint angle continuity.

Cerebella: Automatic Generation of Nonverbal Behavior for Virtual Humans

Our method automatically generates realistic nonverbal performances for virtual characters to accompany spo- ken utterances. It analyses the acoustic, syntactic, se- mantic and rhetorical properties of the utterance text and audio signal to generate nonverbal behavior such as such as head movements, eye saccades, and novel gesture animations based on co-articulation.

Hand gesture recognition and animation for local hand motions

Hand gestures are universally adopted means of communication to convey message in the form of sign language. Therefore, to communicate with a deaf and dumb person, a normal human requires to have some knowledge about the sign language and should be able to make the sign language gestures. By understanding and animating hand gestures, we can help in facilitating communication between computers and the underprivileged. In this paper, we present a method for synthesizing hand gestures with the help of a computer which may enable a normal person to convey massage to a mute person more easily without any knowledge of sign language. The proposed technique requires to train the system prior to its operation. But, gesture animation is computationally complex as it involves replication of the hand with its 27 degrees of freedom. Gesture animation also involves gesture recognition. Hence, in this paper, we have implemented a gesture animation framework after recognizing hand gestures. Computational complexity has been significantly reduced by summarizing large gesture sequence in the form of key frames. The animation process includes hand parameter calculation for every pose in a gesture sequence which is obtained using information like position of fingers, location of metacarpophalangeal joints of the fingers and the bent angles of the fingers. By using these parameters, hand pose estimation is done by imposing some constraints of the hand. Subsequently, a gesture sequence is animated using these models. For this, the hand model for the frames in between the key frames are obtained by interpolation. In our experiment, we demonstrate gesture animation with hand pose exactly same as the real gesture.

Gesture modeling and animation based on a probabilistic re-creation of speaker style

Animated characters that move and gesticulate appropriately with spoken text are useful in a wide range of applications. Unfortunately, this class of movement is very difficult to generate, even more so when a unique, individual movement style is required. We present a system that, with a focus on arm gestures, is capable of producing full-body gesture animation for given input text in the style of a particular performer. Our process starts with video of a person whose gesturing style we wish to animate. A tool-assisted annotation process is performed on the video, from which a statistical model of the person's particular gesturing style is built. Using this model and input text tagged with theme, rheme and focus, our generation algorithm creates a gesture script. As opposed to isolated singleton gestures, our gesture script specifies a stream of continuous gestures coordinated with speech. This script is passed to an animation system, which enhances the gesture description with additional detail. It then generates either kinematic or physically simulated motion based on this description. The system is capable of generating gesture animations for novel text that are consistent with a given performer's style, as was successfully validated in an empirical user study.

Synthesis and Animation of Dynamic Hand Gestures for Sign Language Generation

Synthesis and Animation of Dynamic Hand Gestures for Sign Language Generation

Proposing a speech to gesture translation architecture for Spanish deaf people

This article describes an architecture for translating speech into Spanish Sign Language (SSL). The architecture proposed is made up of four modules: speech recognizer, semantic analysis, gesture sequence generation and gesture playing. For the speech recognizer and the semantic analysis modules, we use software developed by IBM and CSLR (Center for Spoken Language Research at University of Colorado), respectively. Gesture sequence generation and gesture animation are the modules on which we have focused our main effort. Gesture sequence generation uses semantic concepts (obtained from the semantic analysis) associating them with several SSL gestures. This association is carried out based on a number of generation rules. For gesture animation, we have developed an animated agent (virtual representation of a human person) and a strategy for reducing the effort in gesture animation. This strategy consists of making the system automatically generate all agent positions necessary for the gesture animation. In this process, the system uses a few main agent positions (two or three per second) and some interpolation strategies, both issues previously generated by the service developer (the person who adapts the architecture proposed in this paper to a specific domain). Related to this module, we propose a distance between agent positions and a measure of gesture complexity. This measure can be used to analyze the gesture perception versus its complexity. With the architecture proposed, we are not trying to build a domain independent translator but a system able to translate speech utterances into gesture sequences in a restricted domain: railway, flights or weather information.

Representing coordination and non-coordination in American Sign Language animations

While strings and syntax trees are used by the Natural Language Processing community to represent the structure of spoken languages, these encodings are difficult to adapt to a signed language like American Sign Language (ASL). In particular, the multichannel nature of an ASL performance makes it difficult to encode in a linear single-channel string. This paper will introduce the Partition/Constitute (P/C) formalism, a new method of computationally representing a linguistic signal containing multiple channels. The formalism allows coordination and non-coordination relationships to be encoded between different portions of a signal. The P/C formalism will be compared to representations used in related research in gesture animation. The way in which P/C is used by this project to build an English-to-ASL machine translation system will also be discussed.

High-level Specification and Animation of Communicative Gestures

This paper describes a complete system for the specification and the generation of visual communicative gestures. A high-level language for the specification of hand–arm communication gestures was developed. This language is based on both a discrete description of space, and a movement decomposition inspired by sign language gestures. Communication gestures are represented by symbolic commands that can be described by qualitative data, and translated in terms of spatio-temporal targets that drive the generation system. This approach is well-suited for a class of generation models controlled by key-points information at the trajectory level. The animation model used in our approach is composed of a set of sensory-motor control loops. Each of these models computes in real-time updated angular coordinates of the articulatory structure from the minimization of the distance between target and current locations. At the same time, psycho-motor laws of biological movement are satisfied. The whole control system is applied to the synthesis of communication and sign language gestures. A synthetic character is animated and some results are presented.

Interactive Computer Animation of Hand Gestures using Status Estimation with Multiple Regression Analysis

This paper presents a method of interactively generating natural hand gesture animation using reduced dimensionality from multiple captured data sequences of finger motions conducting specific tasks. This method is achieved by introducing an estimation with multiple regression analysis. Even when the skeletal structure of the user who inputs the motion is different from that of the shape model in the computer, the motion that a user imagines is generated. Experimental results obtained from the interface applied to virtual object manipulation showed that the proposed method generates animation naturally, just as users would expect. This method enables us to make input devices that require minimal user training and computer calibration, and helps to make the user interface intuitive and easy to use.

Using Hybrid Automata to Support Human Factors Analysis in a Critical System

A characteristic that many emerging technologies and interaction techniques have in common is a shift towards tighter coupling between human and computer. In addition to traditional discrete interaction, more continuous interaction techniques, such as gesture recognition, haptic feedback and animation, play an increasingly important role. Additionally, many supervisory control systems (such as flight deck systems) already have a strong continuous element. The complexity of these systems and the need for rigorous analysis of the human factors involved in their operation leads us to examine formal and possibly automated support for their analysis. The fact that these systems have important temporal aspects and potentially involve continuous variables, besides discrete events, motivates the application of hybrid systems modelling, which has the expressive power to encompass these issues. Essentially, we are concerned with human-factors related questions whose answers are dependent on interactions between the user and a complex, dynamic system. In this paper we explore the use of hybrid automata, a formalism for hybrid systems, for the specification and analysis of interactive systems. To illustrate the approach we apply it to the analysis of an existing flight deck instrument for monitoring and controlling the hydraulics subsystem.

Pattern Recognition and Synthesis for a Sign Language Translation System

Sign language is one means of communication for hearing-impaired people. Words and sentences in sign language are mainly represented by hand gestures. In this report, we describe a sign language translation system which we are developing. The system translates Japanese Sign Language into Japanese text and vice versa. In this system, hand shapes and positions are provided by hand-based input in the form of time series. Then, hand gestures are recognized and translated into Japanese sentences using continuous DP (dynamic programming) matching. Japanese text is translated into sign language by generating three-dimensional computer-graphic animation of sign language gestures.