Visual Speech Synthesis Research Articles

CCLCap-AE-AVSS: Cycle consistency loss based capsule autoencoders for audio–visual speech synthesis

CCLCap-AE-AVSS: Cycle consistency loss based capsule autoencoders for audio–visual speech synthesis

Research on a Guide Dog Robot for Expressing Visual Environment by Voice

This paper proposed a guide dog robot that can express visual environment by voice. The guide dog robot uses NIC, a combination of CNN and LSTM methods, to recognize the visual scene and generate captions. The generated sentences are then converted to speech using speech synthesis to guide the visually impaired. The guide dog robot can not only guide the visually impaired person safely to their destination, but also providing explanation services of street scenes, enabling the visually impaired person to enjoy visual scenes. The system is consisted of visual scene recognition, caption, and speech synthesis. The effectiveness of the system is confirmed through experiments using the proposed guide dog robot.

Modeling of 3D Human Face for Tamil Visual Speech Synthesis

Modeling of 3D Human Face for Tamil Visual Speech Synthesis

Synthesising visual speech using dynamic visemes and deep learning architectures

This paper proposes and compares a range of methods to improve the naturalness of visual speech synthesis. A feedforward deep neural network (DNN) and many-to-one and many-to-many recurrent neural networks (RNNs) using long short-term memory (LSTM) are considered. Rather than using acoustically derived units of speech, such as phonemes, viseme representations are considered and we propose using dynamic visemes together with a deep learning framework. The input feature representation to the models is also investigated and we determine that including wide phoneme and viseme contexts is crucial for predicting realistic lip motions that are sufficiently smooth but not under-articulated. A detailed objective evaluation across a range of system configurations shows that a combined dynamic viseme-phoneme speech unit combined with a many-to-many encoder-decoder architecture models visual co-articulations effectively. Subjective preference tests reveal there to be no significant difference between animations produced using this system and using ground truth facial motion taken from the original video. Furthermore, the dynamic viseme system also outperforms significantly conventional phoneme-driven speech animation systems.

A Realistic Visual Speech Synthesis for Indonesian Using A Combination of Morphing Viseme and Syllable Concatenation Approach to Support Pronunciation Learning

This study aims to build a realistic visual speech synthesis for Indonesian so that it can be used to learn Indonesian pronunciation. In this study, We used the combination of morphing viseme and syllable concatenation method. The morphing viseme method is a process of deformation from one viseme to another so that the animation of the mouth shape looks smoother. This method is used to create the transition of animation between viseme. The Syllable Concatenation method is used to assemble viseme based on certain syllable patterns. We built a syllable-based voice database as a basis for synchronization between syllables, speech and viseme models. The method proposed in this study consists of several stages, namely the formation of Indonesian viseme models, designing facial animation character, development of speech database, a synchronization process and subjective testing of the resulting application. Subjective tests were conducted on 30 respondents who assessed the suitability and natural movement of the mouth when uttering the Indonesian texts. The MOS (Mean Opinion Score) method is used to calculate the average of respondents' scores. The MOS calculation results for the criteria of Synchronization and naturalness are 4,283 and 4,107 on the scale of 1 to 5. This result shows that the level of Synchronization and naturalness of the synthesis of visual speech is more realistic. Therefore, the system can display the visualization of phoneme pronunciation to support learning Indonesian pronunciation.

Creating and simulating a realistic physiological tongue model for speech production

Simulation of the tongue has important applications in biomechanics, medical science, linguistics, and graphics. The accuracy of the geometry, intrinsic structure and dynamic simulation of tongue are crucial for these applications. In this paper, we build a 3D anatomically and biomechanically accurate tongue model. For ensuring anatomical accuracy, the tongue mesh model is constructed based on accurate medical data and an interactive muscle marking method for specifying the muscle geometry and fiber arrangement. For ensuring biomechanical accuracy, a nonlinear, quasi-incompressible, isotropic, hyperelastic constitutive model is applied for describing the tongue tissues. Particularly, tongue muscles are additionally endowed with an anisotropic constitutive model, which reflects the active and passive mechanical behavior of muscle fibers. The dynamic simulation results of tongue movements subjected to certain muscle activations are presented and validated with experimental data, indicating the suitability for visual speech synthesis.

Describing and Animating Complex Communicative Verbal and Nonverbal Behavior Using Eva-Framework

Multimodal interfaces incorporating embodied conversational agents enable the development of novel concepts with regard to interaction management tactics in responsive human–machine interfaces. Such interfaces provide several additional nonverbal communication channels, such as natural visualized speech, facial expression, and different body motions. In order to simulate reactive humanlike communicative behavior and attitude, the realization of motion relies on different behavioral analyses and realization tactics and approaches. This article proposes a novel environment for “online” visual modeling of humanlike communicative behavior, named EVA-framework. In this study we focus on visual speech and nonverbal behavior synthesis by using hierarchical XML-based behavioral events and expressively adjustable motion templates. The main goal of the presented abstract motion notation scheme, named EVA-Script, is to enable the synthesis of unique and responsive behavior.

Visual Speech Synthesis Using a Variable-Order Switching Shared Gaussian Process Dynamical Model

In this paper, we present a novel approach to speech- driven facial animation using a non-parametric switching state space model based on Gaussian processes. The model is an extension of the shared Gaussian process dynamical model, augmented with switching states. Two talking head corpora are processed by extracting visual and audio data from the sequences followed by a parameterization of both data streams. Phonetic labels are obtained by performing forced phonetic alignment on the audio. The switching states are found using a variable length Markov model trained on the labelled phonetic data. The audio and visual data corresponding to phonemes matching each switching state are extracted and modelled together using a shared Gaussian process dynamical model. We propose a synthesis method that takes into account both previous and future phonetic context, thus accounting for forward and backward coarticulation in speech. Both objective and subjective evaluation results are presented. The quantitative results demonstrate that the proposed method outperforms other state-of-the-art methods in visual speech synthesis and the qualitative results reveal that the synthetic videos are comparable to ground truth in terms of visual perception and intelligibility.

Comprehensive many-to-many phoneme-to-viseme mapping and its application for concatenative visual speech synthesis

The use of visemes as atomic speech units in visual speech analysis and synthesis systems is well-established. Viseme labels are determined using a many-to-one phoneme-to-viseme mapping. However, due to visual coarticulation effects, an accurate mapping from phonemes to visemes should define a many-to-many mapping scheme instead. In this research it was found that neither the use of standardized nor speaker-dependent many-to-one viseme labels could satisfy the quality requirements of concatenative visual speech synthesis. Therefore, a novel technique to define a many-to-many phoneme-to-viseme mapping scheme is introduced, which makes use of both tree-based and k-means clustering approaches. We show that these many-to-many viseme labels more accurately describe the visual speech information as compared to both phoneme-based and many-to-one viseme-based speech labels. In addition, we found that the use of these many-to-many visemes improves the precision of the segment selection phase in concatenative visual speech synthesis using limited speech databases. Furthermore, the resulting synthetic visual speech was both objectively and subjectively found to be of higher quality when the many-to-many visemes are used to describe the speech database and the synthesis targets.

A Kind of Visual Speech Feature with the Geometric and Local Inner Texture Description

In this paper, we propose a type of joint feature with geometric parameters and color moments to represent the speaking-mouth frames for image-based visual speech synthesis systems. Based on FDP around the mouth area, the geometric feature is obtained by computing Euclidean distances to describe the width of the speaking mouth, the height of the outer and inner lips and the distances between them. The color moment component in the joint feature is obtained by calculating the texture between the upper and lower inner lips to describe the visibility state of the teeth. Through analyzing the accordance between the teeth visibility and the components of RGB and HSV color space based on the samples separately, we discovered that green and blue components are good at describing the change of teeth visibility. The experiments show that the proposed joint feature can effectively provide the basis for categorizing the different speaking states especially at the sense of lip shapes and tooth visibility. The evaluation of clustering results is done by analyzing the derived parameters of the silhouette function. The analyzing results prove that comparing with the geometric only and PCA, our proposed feature together with the shape and the local inner lip texture clues has better performance in improving the similarity between samples within the clusters. In the future, more expressive features with the shape and local texture information should be explored to increase the proportion of similar samples within the clusters to improve the descriptive ability of speaking mouths. DOI: http://dx.doi.org/10.11591/telkomnika.v11i2.2047

Relating Objective and Subjective Performance Measures for AAM-Based Visual Speech Synthesis

We compare two approaches for synthesizing visual speech using active appearance models (AAMs): one that utilizes acoustic features as input, and one that utilizes a phonetic transcription as input. Both synthesizers are trained using the same data and the performance is measured using both objective and subjective testing. We investigate the impact of likely sources of error in the synthesized visual speech by introducing typical errors into real visual speech sequences and subjectively measuring the perceived degradation. When only a small region (e.g., a single syllable) of ground-truth visual speech is incorrect we find that the subjective score for the entire sequence is subjectively lower than sequences generated by our synthesizers. This observation motivates further consideration of an often ignored issue, which is to what extent are subjective measures correlated with objective measures of performance? Significantly, we find that the most commonly used objective measures of performance are not necessarily the best indicator of viewer perception of quality. We empirically evaluate alternatives and show that the cost of a dynamic time warp of synthesized visual speech parameters to the respective ground-truth parameters is a better indicator of subjective quality.

Face Detection Based on Statistical Color Model and Haar Classifier

The paper realizes the face detection algorithm based on the combination of the skin model and the Haar algorithm. Firstly, a platform for sample labeling was constructed, which combines the contour extraction algorithm with manual labeling. By labeling more than 10000 images obtained randomly from the Internet, a large training dataset is available. Then, a skin histogram, a non-skin histogram and a statistical skin model are constructed by analyzing the distribution of the skin and the non-skin color on the basis of a large training dataset. Based on this statistical color model, the skin area is detected and split from video files frame by frame. With the Haar Object Detection algorithm and the morphology algorithm such as erosion and dilation, the background noise and non-face areas are removed from the detected skin area and facial area is detected, which provides the basis for face recognition and the video-based visual speech synthesis. Compared with the Haar-based face detection method, our algorithm greatly improves the rate of correct detection and reduces the rate of the false positives.

Dynamic Chinese viseme model based on phones and control function

The aspirate and pronunciation of a Chinese syllable is similar to an `olivary nucleus` and syllables are mildly connected by coarticulation. Accordingly a dynamic viseme model is presented. Inner-syllable visemes are described by initial-final phones, and inter-syllable visemes are determined by a hierarchical control function. Experimental results show improvement in naturalness of lip animation, as compared to the visual speech synthesis system based on triphones.

A 3-D Audio-Visual Corpus of Affective Communication

Communication between humans deeply relies on the capability of expressing and recognizing feelings. For this reason, research on human-machine interaction needs to focus on the recognition and simulation of emotional states, prerequisite of which is the collection of affective corpora. Currently available datasets still represent a bottleneck for the difficulties arising during the acquisition and labeling of affective data. In this work, we present a new audio-visual corpus for possibly the two most important modalities used by humans to communicate their emotional states, namely speech and facial expression in the form of dense dynamic 3-D face geometries. We acquire high-quality data by working in a controlled environment and resort to video clips to induce affective states. The annotation of the speech signal includes: transcription of the corpus text into the phonological representation, accurate phone segmentation, fundamental frequency extraction, and signal intensity estimation of the speech signals. We employ a real-time 3-D scanner to acquire dense dynamic facial geometries and track the faces throughout the sequences, achieving full spatial and temporal correspondences. The corpus is a valuable tool for applications like affective visual speech synthesis or view-independent facial expression recognition.

Emphatic Visual Speech Synthesis

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> The synthesis of talking heads has been a flourishing research area over the last few years. Since human beings have an uncanny ability to read people's faces, most related applications (e.g., advertising, video-teleconferencing) require absolutely realistic photometric and behavioral synthesis of faces. This paper proposes a person-specific facial synthesis framework that allows high realism and includes a novel way to control visual emphasis (e.g., level of exaggeration of visible articulatory movements of the vocal tract). There are three main contributions: a geodesic interpolation with visual unit selection, a parameterization of visual emphasis, and the design of minimum size corpora. Perceptual tests with human subjects reveal high realism properties, achieving similar perceptual scores as real samples. Furthermore, the visual emphasis level and two communication styles show a statistical interaction relationship. </para>

Realistic Visual Speech Synthesis Based on Hybrid Concatenation Method

This paper presents a realistic visual speech synthesis based on the hybrid concatenation method. Unlike previous methods based on phoneme level unit selection or hidden Markov model (HMM), etc., the hybrid concatenation method uses a frame level-based unit selection method combined with a fused HMM, and is able to generate more expressive and stable facial animations. The fused HMM can be used to explicitly model the loose synchronization of tightly coupled streams, with much better results than a normal HMM for audiovisual mapping. After fused HMM is created, facial animation is generated via the unit selection method at the frame level by using the fused HMM output probabilities. To accelerate the computing efficiency of the unit selection on a large corpus, this paper also proposes a two-layer Viterbi search method in which only the subsets that have been selected in the first layer are further checked in the second layer. Using this idea, the system has been successfully integrated into real-time applications. Furthermore, the paper also proposes a mapping method to generate emotional facial expressions from neutral facial expressions based on Gaussian mixture models (GMMs). Final experiments prove that the method described can output synthesized facial parameters with high quality. Compared with other audiovisual mapping methods, our method has better performance with respect to expressiveness, stability, and system running speed.

Animating Virtual Speakers or Singers from Audio: Lip-Synching Facial Animation

The aim of this special issue is to provide a detailed description of state-of-the-art systems for animating faces during speech, and identify new techniques that have recently emerged from both the audiovisual speech and computer graphics research communities. This special issue is a followup to the first LIPS Visual Speech Synthesis Challenge held as a special session at INTERSPEECH 2008 in Brisbane, Australia. As a motivation for the present special issue, we will report on the LIPS Challenge with respect to the synthesis techniques, andmore importantly the methods and results of the subjective evaluation.

Design, implementation and evaluation of the Czech realistic audio-visual speech synthesis

This paper presents the whole process of creation of audio-visual speech synthesis system. Such system consists of two main parts, the acoustic synthesis emulating human speech and the facial animation emulating the human lip articulation. The acoustic subsystem is based on concatenation-based speech synthesis. The visual subsystem is designed as a realistic, fully three-dimensional parametrically controllable facial animation model. To be able to parametrically control the animation to emulate human articulation, the set of visual parameters has to be obtained for all basic speech units. To provide realistic animation, the database of lip movements of a real person need to be recorded and expressed by suitable parameterization. The set of control parameters for visual animation is then derived from this database. The 3D model of a head based on a head of a real person also makes the animation more realistic. To obtain such model, a 3D scanning of a real person has to be adopted. We present the design and implementation of above-mentioned process. The aim is to obtain realistic audio-visual speech synthesis with possibility to change the 3D head model according to particular person. The design, acquisition and processing of audio-visual speech corpus for such purpose is presented. Next, the process of both acoustic and visual speech synthesis is described. The visual speech synthesis comprises the tasks of model training, animation control, and co-articulation modelling. A facial animation can also increase intelligibility of a telephone speech to people with hearing disabilities. In such case the textual information to control the animation is not available. Solution to the problem of mapping visual parameters from speech signal either directly or through recognized text is presented. Furthermore, the 3D scanning algorithm is presented. It allows to obtain realistic 3D model based on a head of a real person and thus to personalize the talking head. In the end of this paper, evaluation of intelligibility of the presented audio-visual speech synthesis and its possible applications are presented.

Data-driven multimodal synthesis

This paper is a report on current efforts at the Department of Speech, Music and Hearing, KTH, on data-driven multimodal synthesis including both visual speech synthesis and acoustic modeling. In the research we try to combine both corpus based methods with knowledge based models and to explore the best of the two approaches. In the paper an attempt to build formant-synthesis systems based on both rule-generated and database driven methods is presented. A pilot experiment is also reported showing that this approach can be a very interesting path to explore further. Two studies on visual speech synthesis are reported, one on data acquisition using a combination of motion capture techniques and one concerned with coarticulation, comparing different models.

Phoneme clustering based on segmental lip configurations in naturally spoken sentences

It has been shown that face (lips, cheeks, and chin) information can account to a large extent for visual speech perception in isolated syllables and words. Visual speech synthesis has used small reduced sets of phonemes (‘‘visemes’’), under the theory that perceivers are limited in their ability to extract visual speech information. In this study, lip configurations from a manually segmented sentence database [L. Bernstein et al., J. Acoust. Soc. Am. 107, 2887 (2000)] were analyzed to provide phoneme clusters that are algorithmically distinguishable using mouth vertical/horizontal opening and lip protrusion from the middle position of each segment. The lip feature sample spaces for each phoneme were represented by Gaussian mixture models. Maximum posterior probability classification results were computed for each phoneme. Confusion matrices were generated from the classification results, and a set of confusions with 74% or higher within-group classification correct was judged to be a cluster. Preliminary results from 191 sentences by a single talker generated the following clusters: {/p, b, m/(77%), /f, v/(74%), /w, r/(80%), /t, d, s, z, D, k, n/(88%)}. We will present results analyzing the entire English phoneme set across different talkers and compare the results with visual perceptual clusters. [Work supported in part by the NSF.]