Facial Image Synthesis Research Articles

Synthesis of Facial Image with Expression Based on Muscular Contraction Parameters Using Linear Muscle and Sphincter Muscle

We aim to synthesize individual facial image with expression based on muscular contraction parameters. We have proposed a method of calculating the muscular contraction parameters from arbitrary face image without using learning for each individual. As a result, we could generate not only individual facial expression, but also the facial expressions of various persons. In this paper, we propose the muscle-based facial model; the facial muscles define both the linear and the novel sphincter. Additionally, we propose a method of synthesizing individual facial image with expression based on muscular contraction parameters. First, the individual facial model with expression is generated by fitting using the arbitrary face image. Next, the muscular contraction parameters are calculated that correspond to the expression displacement of the input face image. Finally, the facial expression is synthesized by the vertex displacements of a neutral facial model based on calculated muscular contraction parameters. Experimental results reveal that the novel sphincter muscle can synthesize facial expressions of the facial image, which corresponds to the actual face image with arbitrary and mouth or eyes expression.

가상대학에서 교수자와 학습자간 상호작용을 위한 지식기반형 문자-얼굴동영상 변환 시스템

본 논문에서는 가상대학에서 교수자와 학습자간 상호작용을 위한 지식기반형 문자-얼굴동영상 변환(TTFSI : Text to Facial Sequence Image) 시스템에 관해 연구하였다. TTFSI 시스템의 구현을 위해, 한글의 문법적 특징을 기반으로 가상강의에 사용된 자막정보에 립싱크된 얼굴 동영상 합성하기 위하여 자막정보를 음소코드로 변환하는 방법, 음소코드별 입모양의 변형규칙 작성법, 입모양 변형규칙에 의한 얼굴 동영상 합성법을 제안한다. 제안된 방법에서는 한글의 구조분석을 통해 기본 자모의 발음을 나타내는 10개의 대표 입모양과 조음결합에서 나타나는 78개의 혼합 입모양으로 모든 음절의 입모양을 표현하였다. 특히 PC환경에서의 실시간 영상을 합성하기 위해서 매 프레임마다 입모양을 합성하지 않고, DB에서 88개의 해당 입모양을 불러오는 방법을 사용하였다. 제안된 방법의 유용성을 확인하기 위하여 텍스트 정보에 따른 다양한 얼굴 동영상을 합성하였으며, PC환경에서 구현 가능한 TTFSI 시스템을 구축하였다. In this paper, knowledge based text to facial sequence image system for interaction of lecturer and learner in cyber universities is studied. The system is defined by the synthesis of facial sequence image which is synchronized the lip according to the text information based on grammatical characteristic of hangul. For the implementation of the system, the transformation method that the text information is transformed into the phoneme code, the deformation rules of mouse shape which can be changed according to the code of phonemes, and the synthesis method of facial sequence image by using deformation rules of mouse shape are proposed. In the proposed method, all syllables of hangul are represented 10 principal mouse shape and 78 compound mouse shape according to the pronunciation characteristics of the basic consonants and vowels, and the characteristics of the articulation rules, respectively. To synthesize the real time facial sequence image able to realize the PC, the 88 mouth shape stored data base are used without the synthesis of mouse shape in each frame. To verify the validity of the proposed method the various synthesis of facial sequence image transformed from the text information is accomplished, and the system that can be applied the PC is implemented using the proposed method.

Affect Display Effects of Anger by Virtual Facial Image Synthesis with Facial Color Enhancement and Expression

In this paper, we demonstrate the effects of the affect display of anger by virtual facial image synthesis with facial color enhancement and expression on the basis of the analysis of facial color under the emotional change of anger. First, we examine the relationship between the change in the facial color and the physiological index of facial skin temperature, which indicates the change in the blood flux under the emotional change of anger. Second, on the basis of this result, we synthesize 11 enhanced facial color images by using an average facial image of 68 females with their eyes and mouth masked out ; we evaluate the effectiveness of these images through a sensory evaluation by performing a paired comparison. Further, we synthesize enhanced facial color images by using the average facial image in which only the eyes were masked out and evaluate the correspondence between the lip color and facial color after performing the enhanced facial color synthesis for an affect display of facial color through a sensory evaluation. Finally, we synthesize the facial expression of anger with enhanced facial color by using the average facial image and evaluate the effectiveness for the affect display of the image through the sensory evaluation of seven-point bipolar ratings.

Analysis and Synthesis of Facial Color for Facial Image Synthesis in a Virtual Arm Wrestling System

An arm wrestling system using a 4-DOF force display with a virtual human opponent is developed, incorporating affect display of facial color and expression, targeting training and rehabilitation support applications. First, a 4-air cylinder 4-DOF force display system is developed for arm wrestling with human operators. Using this system, changes in facial skin temperature as an index of emotion and facial color associated with circulation dynamics in response to the force display, are simultaneously measured and analyzed. Second, based on this analysis, a synthetic method for affect display of the virtual face using facial color and expression for use with the force is proposed, demonstrating its effectiveness through sensory evaluation. Finally, the results are combined to develop an arm wrestling prototype with a virtual human opponent. This paper is the full translation from the transactions of JSME Series C, Vol.70, No.693, 2004.

Representation Analysis and Synthesis of Lip Images Using Dimensionality Reduction

Understanding facial expressions in image sequences is an easy task for humans. Some of us are capable of lipreading by interpreting the motion of the mouth. Automatic lipreading by a computer is a challenging task, with so far limited success. The inverse problem of synthesizing real looking lip movements is also highly non-trivial. Today, the technology to automatically generate an image series that imitates natural postures is far from perfect.We introduce a new framework for facial image representation, analysis and synthesis, in which we focus just on the lower half of the face, specifically the mouth. It includes interpretation and classification of facial expressions and visual speech recognition, as well as a synthesis procedure of facial expressions that yields natural looking mouth movements.Our image analysis and synthesis processes are based on a parametrization of the mouth configuration set of images. These images are represented as points on a two-dimensional flat manifold that enables us to efficiently define the pronunciation of each word and thereby analyze or synthesize the motion of the lips. We present some examples of automatic lips motion synthesis and lipreading, and propose a generalization of our solution to the problem of lipreading different subjects.

Matching 2.5D face scans to 3D models

The performance of face recognition systems that use two-dimensional images depends on factors such as lighting and subject's pose. We are developing a face recognition system that utilizes three-dimensional shape information to make the system more robust to arbitrary pose and lighting. For each subject, a 3D face model is constructed by integrating several 2.5D face scans which are captured from different views. 2.5D is a simplified 3D (x, y, z) surface representation that contains at most one depth value (z direction) for every point in the (x, y) plane. Two different modalities provided by the facial scan, namely, shape and texture, are utilized and integrated for face matching. The recognition engine consists of two components, surface matching and appearance-based matching. The surface matching component is based on a modified Iterative Closest Point (ICP) algorithm. The candidate list from the gallery used for appearance matching is dynamically generated based on the output of the surface matching component, which reduces the complexity of the appearance-based matching stage. Three-dimensional models in the gallery are used to synthesize new appearance samples with pose and illumination variations and the synthesized face images are used in discriminant subspace analysis. The weighted sum rule is applied to combine the scores given by the two matching components. Experimental results are given for matching a database of 200 3D face models with 598 2.5D independent test scans acquired under different pose and some lighting and expression changes. These results show the feasibility of the proposed matching scheme.

2112 表情と顔色の色彩強調に伴うバーチャル顔画像合成による怒りの情動提示効果(OS1,13 ヒューマンインタフェース・ユーザビリティ,未来社会を支えるものづくりとひとづくり(設計・システムから))

2112 表情と顔色の色彩強調に伴うバーチャル顔画像合成による怒りの情動提示効果(OS1,13 ヒューマンインタフェース・ユーザビリティ,未来社会を支えるものづくりとひとづくり(設計・システムから))

Facial expressional image synthesis controlled by emotional parameters

Facial expressional image synthesis is an important technique in human–computer interaction, but it is also a difficult task, especially to make various realistic expressive face images with a flexible control mechanism. In this paper, a novel parameter driven method of synthesizing realistic comprehensive expressional image is proposed. In this method a kernel-based bi-factor factorization model is adopted to decompose two influence factors, identity and expression, and their interaction from a small training database. Therefore the facial expressional images in the training database can be covered by corresponding identity and expression vectors and their interaction matrix. Comprehensive expression image can be manipulated by means of linear combination of these basis expression vectors in a flexible manner. In order to make this trained model be capable of producing realistic expressional images of any person outside the training set, expression ratio image (ERI) and relative shape description are joined into our model to enhance the expressive power of our method. Experimental results show that realistic facial expressional image can be synthesized successfully from only one picture of someone who is quite different from persons in the training database, and controlled by a parameter vector efficiently and effectively.

Analysis and Synthesis of Facial Color for Facial Image Synthesis in a Virtual Arm Wrestling System

An arm wrestling system using a 4-DOF force display with virtual human is developed for the affect display of facial color and expression, aiming for the application to training and rehabilitation support. First, the 4-DOF force display system with 4 air cylinders is developed for arm wrestling with human. By using the system, the changes of facial skin temperature as an index of emotional state and facial color associated with circulation dynamics in response to force display are simultaneously measured and analyzed. Second, on the basis of the analysis, a synthetic method for the affect display of virtual face with facial color and expression in force display is proposed, and sensory evaluation demonstrates the effectiveness of the method. Finally, by using the method, a prototype of arm wrestling system with virtual human is developed.

Reconstruction and analysis of multi-pose face images based on nonlinear dimensionality reduction

Locally linear embedding (LLE) is a nonlinear dimensionality reduction method proposed recently. It can reveal the intrinsic distribution of data, which cannot be provided by classical linear dimensionality reduction methods. The application of LLE, however, is limited because of its lack of a parametric mapping between the observation and the low-dimensional output. And the large data set to be reduced is necessary. In this paper, we propose methods to establish the process of mapping from low-dimensional embedded space to high-dimensional space for LLE and validate their efficiency with the application of reconstruction of multi-pose face images. Furthermore, we propose that the high-dimensional structure of multi-pose face images is similar for the same kind of pose change mode of different persons. So given the structure information of data distribution which is obtained by leaning large numbers of multi-pose images in a training set, the support vector regression (SVR) method of statistical learning theory is used to learn the high-dimensional structure of someone based on small sets. The detailed learning method and algorithm are given and applied to reconstruct and synthesize face images in small set cases. The experiments prove that our idea and method is correct.

Guest editorial: Facial image processing, analysis and synthesis

Guest editorial: Facial image processing, analysis and synthesis

Interactive evolutionary computation: fusion of the capabilities of EC optimization and human evaluation

We survey the research on interactive evolutionary computation (IEC). The IEC is an EC that optimizes systems based on subjective human evaluation. The definition and features of the IEC are first described and then followed by an overview of the IEC research. The overview primarily consists of application research and interface research. In this survey the IEC application fields include graphic arts and animation, 3D computer graphics lighting, music, editorial design, industrial design, facial image generation, speed processing and synthesis, hearing aid fitting, virtual reality, media database retrieval, data mining, image processing, control and robotics, food industry, geophysics, education, entertainment, social system, and so on. The interface research to reduce human fatigue is also included. Finally, we discuss the IEC from the point of the future research direction of computational intelligence. This paper features a survey of about 250 IEC research papers.

ヒューマンインタフェース 合成顔画像を用いた脳障害の程度評価

ヒューマンインタフェース 合成顔画像を用いた脳障害の程度評価

Synthesis and transformation of three-dimensional facial images.

In this article, we enhance and extend 2-D methods by using texture-mapped laser-video depth data. Methods for the synthesis of new three-dimensional (3-D) facial images from a database of texture-mapped laser-scanned surfaces are described. Four methods are used to create new facial images. The first constructs average facial images from a sample of faces. The second exaggerates the difference between a subject and an average to create a caricature. The third method captures the difference between two averages, which can then be applied to individuals for statistical transformation. Finally, principal component analysis is used to find the major axes of facial variation in a sample, allowing the construction of completely novel faces from a small vector of values.

Facial Color Image Analysis and Synthesis for the Virtual Face Image in Emotional Change.

Human facil color can be applied to the display of virtual face image, emotional evaluation, individual identification, and remote health care. We already developed a method for emphasizing the individual characteristics in facial color based on a virtual face image using the HSV (hue, saturation, value) color space. To make virtual face images more natural and richer, the clarification of the relation between facial color and emotion would be expected. In this paper, we propose a synthetic method for the virtual face image based on the analysis of facial color in emotional change. First, the facial colors under the emotion of joy or anger are determined quantitatively by using the HSV color space. Secondly, we examined the relation between the facial color and the physiological index of facial skin temperature, which indicates the change of blood flux, in drinking for the emotion of joy. Also, we examined the relation between the change of facial color and that of skin temperature under the joy in laughing. Finally, on the basis of these results, we synthesized seven enhanced facial color images based on an average facial color image from 68 females, and evaluated the effectiveness of these images through sensory evaluation by paired comparison.

Processing of facial information by computer

Faces are very familiar to everyone and convey various information, including information that is specific to the individual and information that is part of mutual communication. Usually verbal media is not able to describe such information appropriately. Recently, detailed studies on facial information processing by computer have been carried out in the engineering field for application to communication media and human interfaces. Two main topics are recognition of human faces and synthesis of facial images. The objective of the former is to enable computers to recognize human faces and that of the latter is to provide a natural and impressive interface in the form of a “face” for communication media. These technologies have also been found to be useful in various fields related to the face, such as psychology, anthropology, cosmetology, and dentistry. Although most of the studies in these fields have been carried out independently, the above technologies provide for a common study tool among them. This paper focuses on processing facial information by computer. First, recent research trends on the synthesis of facial images and recognition of facial expressions are outlined. After considering the various characteristics of the face, the engineering applications of facial information are discussed from two standpoints: the support of face-to-face communication between humans, and communication between human and machine using facial information. The tools and databases used in studying facial information processing and some related topics are also described. © 1998 Scripta Technica, Electron Comm Jpn Pt 3, 81(10): 36–57, 1998

An X Windows tool for synthesizing face images from eigenvectors

An X Windows software tool for the construction of faces with a weighted combination of eigenvectors is described. The eigenvectors were extracted from an autoassociative matrix that comprised 100 face images. The program input consists of eigenvectors and sets of weights that describe individual faces and combines these to create face images. The tool creates a panel of buttons that permits the display of individual eigenvectors and the display of an average face as well. Facilities for on-line changes to the intensity of individual eigenvectors can be used to change the appearance of a face. Previously, O’Toole, Abdi, Deffenbacher, and Bartlett (1991) have shown that the intensity of certain individual eigenvectors contains reliable information for determining the sex and race of the face.

A realtime model-based facial image synthesis based on multiprocessor network

Model-based image coding has been highlighted recently as a high-efficiency coding method for TV telephone and TV conference systems. In a model-based coding system, an ultralow-rate image transmission is realized by obtaining common models of a facial image at both sides of a communication and by transmitting only modification parameters between them. However, it is difficult to realize a realtime processing of a model-based coding with a conventional iterative-processing type computer since the amount of material to analyze and synthesize at both sides is very large. Realtime processing is absolutely necessary to realize a practical system using this method, i.e., highspeed processings at both the transmitter and receiver are required. This paper describes a realtime facial image synthesis method based on a multiprocessor construction with a transputer. The transputer is a microcomputer having a communication capability with multiple processors and 10 MIPS CPU performance. Using this system in a pipeline processing with 20 processors, it is possible to synthesize realtime facial images at a rate of about 16 frames per second. If only a part around the lips is transmitted, it is possible to synthesize an image at a rate of about 40 frames per second with a network using five processors.

A facial motion synthesis for intelligent man‐machine interface

AbstractA facial motion image synthesis method for intelligent man‐machine interface is examined. Here, the intelligent man‐machine interface is a kind of friendly man‐machine interface with voices and pictures in which human faces appear on a screen and answer questions, compared to the currently existing user interfaces which primarily uses letters. Thus what appears on the screen is human faces, and if speech mannerisms and facial expressions are natural, then the interactions with the machine are similar to those with actual human beings.To implement such an intelligent man‐machine interface it is necessary to synthesize natural facial expressions on the screen. This paper investigates a method to synthesize facial motion images based on given information on text and emotion. The proposed method utilizes the analysis‐synthesis image coding method. It constructs facial images by assigning intensity data to the parameters of a 3‐dimensional (3‐D) model matching the person in question. Moreover, it synthesizes facial expressions by modifying the 3‐D model according to the predetermined set of rules based on the input phonemes and emotion, and also synthesizes reasonably natural facial images.

Facial image synthesis using skin texture recording

A significant issue in synthesizing realistic faces is the representation of skin grain and wrinkles. This paper describes a new approach based on the registered body of a real person. This recording allows simultaneously registering of the 3-D coordinates of a point and the corresponding reflected intensity. Using a 4-D B-spline surface to reconstruct the face, we come close to achieving a photographic result.