Real-time Facial Expression Research Articles

Training facial expression production in children on the autism spectrum.

Children with autism spectrum disorder (ASD) show deficits in their ability to produce facial expressions. In this study, a group of children with ASD and IQ-matched, typically developing (TD) children were trained to produce "happy" and "angry" expressions with the FaceMaze computer game. FaceMaze uses an automated computer recognition system that analyzes the child's facial expression in real time. Before and after playing the Angry and Happy versions of FaceMaze, children posed "happy" and "angry" expressions. Naïve raters judged the post-FaceMaze "happy" and "angry" expressions of the ASD group as higher in quality than their pre-FaceMaze productions. Moreover, the post-game expressions of the ASD group were rated as equal in quality as the expressions of the TD group.

Muecas: a multi-sensor robotic head for affective human robot interaction and imitation.

This paper presents a multi-sensor humanoid robotic head for human robot interaction. The design of the robotic head, Muecas, is based on ongoing research on the mechanisms of perception and imitation of human expressions and emotions. These mechanisms allow direct interaction between the robot and its human companion through the different natural language modalities: speech, body language and facial expressions. The robotic head has 12 degrees of freedom, in a human-like configuration, including eyes, eyebrows, mouth and neck, and has been designed and built entirely by IADeX (Engineering, Automation and Design of Extremadura) and RoboLab. A detailed description of its kinematics is provided along with the design of the most complex controllers. Muecas can be directly controlled by FACS (Facial Action Coding System), the de facto standard for facial expression recognition and synthesis. This feature facilitates its use by third party platforms and encourages the development of imitation and of goal-based systems. Imitation systems learn from the user, while goal-based ones use planning techniques to drive the user towards a final desired state. To show the flexibility and reliability of the robotic head, the paper presents a software architecture that is able to detect, recognize, classify and generate facial expressions in real time using FACS. This system has been implemented using the robotics framework, RoboComp, which provides hardware-independent access to the sensors in the head. Finally, the paper presents experimental results showing the real-time functioning of the whole system, including recognition and imitation of human facial expressions.

Hyperscanning MEG for understanding mother–child cerebral interactions

Child development is seriously affected by social interactions with caregivers, which may lead to forming social minds in our daily life afterward. However, the underlying neural mechanism for such interactions has not yet been revealed. This article introduces a magnetoencephalographic (MEG) hyperscanning system to examine brain-to-brain interactions between a mother and her child. We used two whole-head MEG systems placed in the same magnetically-shielded room. One is a 160-channel gradiometer system for an adult and the other is a 151-channel gradiometer system for a child. We developed an audio-visual presentation system, which enabled a mother and her child to look at each other in real time. In each MEG system, a video camera was placed behind a half-mirror screen for visual presentation to obtain the subjects' facial expressions. The visual presentation system is capable of displaying not only real-time facial expression but also processed facial expression such as a still face or delayed facial expressions. A projector system displays the side-by-side face images of the mother and child, and the images are divided into each face using splitting mirrors and each face is displayed on the half-mirror screen in front of the other subject. To the best of our knowledge, our system is the first MEG hyperscanning system in a single shielded room, and may contribute to elucidating brain-to-brain interactions not only between a mother and her child but also in general inter-individual, brain-to-brain interactions.

An approach to facial expression analysis with multi-model interactions

Affective computing is important to the human–computer interaction, while facial expression analysis is the core issue. In this paper, a multi-model approach is proposed to achieve rational and satisfactory results for real-time facial expression analysis. With the advantages of multi-model interaction, the traditional process of cluster construction is optimized for the facial expression cluster structures according to the multi-model selection, distribution and evaluation interactions. Experiments are conducted to evaluate the rationality of the multi-model approach outlined in this paper. The satisfactory results demonstrate favourable performance comparable to the best results achieved through the cooperative neuron-computing interactions. Not only can the resultant approach construct the cluster distribution efficiently and accurately, but also it is capable of achieving high-quality and high-convergence interactive computing.

Framework for reliable, real-time facial expression recognition for low resolution images

Automatic recognition of facial expressions is a challenging problem specially for low spatial resolution facial images. It has many potential applications in human–computer interactions, social robots, deceit detection, interactive video and behavior monitoring. In this study we present a novel framework that can recognize facial expressions very efficiently and with high accuracy even for very low resolution facial images. The proposed framework is memory and time efficient as it extracts texture features in a pyramidal fashion only from the perceptual salient regions of the face. We tested the framework on different databases, which includes Cohn–Kanade (CK+) posed facial expression database, spontaneous expressions of MMI facial expression database and FG-NET facial expressions and emotions database (FEED) and obtained very good results. Moreover, our proposed framework exceeds state-of-the-art methods for expression recognition on low resolution images.

Realistic Real-time Facial Expressions Animation via 3D Morphing Target

Realistic facial animation can enhance the immersion characteristic of 3D games. This paper proposes a 3D morphing target method based on GPU for real-time animation of facial expressions. We employ texture mapping to obtain a realistic appearance, and design a fast morphing process to achieve a relatively high FPS by modifying the morph data structure and implementing the morphing algorithm in the shader. The algorithm can be realized and divided into seven steps including creating morph target expressions, evaluating difference vector, initializing morph data structure, loading data into VBO, linking attributes to shader and rendering facial expressions. The rendering system also gives us convenient interaction with the digital character and realizes quick expressions shift for practical application. Experiment results show that the proposed approach yields realistic real-time expressions animation.

Helping disabled people: the potentials of biometric information

PurposeThe purpose of this paper is to investigate an idea of producing an assistive and augmentative communication (AAC) tool that uses natural human computer interfacing to accommodate the disabilities of children with cerebral palsy (CP) and assist them in their daily communication.Design/methodology/approachThe authors developed a prototype that recognizes the real time detected emotions display on the face and send alerts to the caretakers through Short Messaging System (SMS) or loud speaker.FindingsThe evaluation result shows that the proposed prototype recognizes real time facial expressions from the children with CP with an average of 79.4 per cent, and a maximum of 88.3 per cent (standard deviation of 7.4 per cent) on ten children with CP. Evaluations were also conducted to investigate the effectiveness of the prototype to deliver critical expression messages to their caretakers. The result showed that 98.5 per cent of SMS was sent successfully to the caretakers (pre‐defined mobile phone number) with an average waiting time of 8.3 seconds.Originality/valueThe paper demonstrates the potential of the proposed prototype to assist children with CP to communicate with their caretakers in real time.

Graphical Facial Expression Analysis and Design Method: An Approach to Determine Humanoid Skin Deformation

Abstract The architecture of human face is complex consisting of 268 voluntary muscles that perform coordinated action to create real-time facial expression. In order to replicate facial expression on humanoid face by utilizing discrete actuators, the first and foremost step is the identification of a pair of origin and sinking points (SPs). In this paper, we address this issue and present a graphical analysis technique that could be used to design expressive robotic faces. The underlying criterion in the design of faces being deformation of a soft elastomeric skin through tension in anchoring wires attached on one end to the skin through the sinking point and the other end to the actuator. The paper also addresses the singularity problem of facial control points and important phenomena such as slacking of actuators. Experimental characterization on a prototype humanoid face was performed to validate the model and demonstrate the applicability on a generic platform.

Bursting with anxiety: Adult social referencing in an interpersonal Balloon Analogue Risk Task (BART).

This study developed an interpersonal modification of the balloon analogue risk task (BART) to assess effects of a friend's real-time facial expressions on coparticipants' risk-taking. Twenty pairs of male friends and 20 pairs of female friends completed two counterbalanced versions of BART, one framed in terms of monetary gain, and the other framed in terms of avoiding loss. Each pair included a player who performed the task across both trials and a reference person in a separate cubicle connected via a soundless video link. In the suppression condition, reference persons were told to minimize facial expressions of anxiety as balloons inflated. In the expression condition, they were told to freely express anxiety. As predicted, players took greater risks and burst more balloons in the suppression condition, but only under the gain frame. Players' BART scores across both frames were also significantly negatively correlated with reference persons' scores on a questionnaire measure of dispositional expressivity (BEQ), confirming that other people's expressions can moderate one's own risk-taking.

Application of an Improved Mean Shift Algorithm in Real-time Facial Expression Recognition

In real-time facial expression recognition, accurate and fast face tracking is a very important preparatory part to obtain the image sequence s of facial expressions. For this problem, an improved mean shift algorithm is proposed for real-time face tracking . F acial expression image sequence s are obtained with the method. The method is based on using pixel gray value distribution as the feature as well as combination of the density distribution of the objective gradient direction . A lternating iterative operation s can be carried through the iterative formula of these two features, and thus , we can make the human face rotation and translation movement tracking better . Then we used the geometric model based on human face to locate the region of facial expression features, and estimate the optical flow to calculate the Eigen-flow vectors. Finally , hidden semi-Markov model is used for facial expression recognition s . Experiments show that the proposed method can effectively track the face under rotation and translation movement of the head and it is very effective to obtain the facial expression image sequence s quickly and accurately.

An embedded system for real-time facial expression recognition based on the extension theory

This paper presents a novel facial expression recognition scheme based on extension theory. The facial region is detected and segmented by using feature invariant approaches. Accurate positions of the lips are then extracted as the features of a face. Next, based on the extension theory, basic facial expressions are classified by evaluating the correlation functions among various lip types and positions of the corners of the mouth. Additionally, the proposed algorithm is implemented using the XScale PXA270 embedded system in order to achieve real-time recognition for various facial expressions. Experimental results demonstrate that the proposed scheme can recognize facial expressions precisely and efficiently.

Effects of damping head movement and facial expression in dyadic conversation using real–time facial expression tracking and synthesized avatars

When people speak with one another, they tend to adapt their head movements and facial expressions in response to each others' head movements and facial expressions. We present an experiment in which confederates' head movements and facial expressions were motion tracked during videoconference conversations, an avatar face was reconstructed in real time, and naive participants spoke with the avatar face. No naive participant guessed that the computer generated face was not video. Confederates' facial expressions, vocal inflections and head movements were attenuated at 1 min intervals in a fully crossed experimental design. Attenuated head movements led to increased head nods and lateral head turns, and attenuated facial expressions led to increased head nodding in both naive participants and confederates. Together, these results are consistent with a hypothesis that the dynamics of head movements in dyadicconversation include a shared equilibrium. Although both conversational partners were blind to the manipulation, when apparent head movement of one conversant was attenuated, both partners responded by increasing the velocity of their head movements.

Towards a portable intelligent facial expression recognizer

Facial expression recognition is a challenging task. A facial expression is fonned by contracting or relaxing different facial muscles on human face which results in temporally deformed facial features like wide open mouth, raising eyebrows or etc. Such a system presents challenges. For instances, lighting condition is a very difficult problem to constraint and regulate. On the other hand, real-time processing is also a challenging problem since there are so many facial features to be extracted and processed and sometime conventional classifiers are not even effective to handle those features and then produce good classification perfonnance. This paper discusses the issues on how the advanced feature selection techniques together with good classifiers can playa vital important role of real-time facial expression recognition. The content of this paper is a way to open-up a discussion about building a real-time system to read and respond to the emotions of people from facial expressions.

Real-time facial expression recognition using STAAM and layered GDA classifier

This paper proposes a real-time person independent facial expression recognition in two parts: one is a model fitting part using a proposed stereo active appearance model (STAAM) and another is a person independent facial expression recognition using a layered generalized discriminant analysis (GDA) classifier. The STAAM fitting algorithm uses multiple calibrated perspective cameras to compute the 3D shape and rigid motion parameters. The use of calibration information reduces the number of model parameters, restricts the degree of freedom in the model parameters, and increases the accuracy and speed of fitting. The STAAM uses a modified simultaneous update fitting method that reduces the fitting computation greatly. Also, the layered GDA classifier combines 3D shape and 2D appearance to improve the recognition performance of person independent facial expressions. Experimental results show that (1) the STAAM shows a better fitting stability than the existing multiple-view AAM (MVAAM), (2) the modified simultaneous update algorithm accelerates the AAM fitting speed, and (3) the combination of the 3D shape and 2D appearance features using a layered GDA classifier improves the performance of facial expression recognition greatly.

A Vision-based Approach for Facial Expression Cloning by Facial Motion Tracking

This paper presents a novel approach for facial motion tracking and facial expression cloning to create a realistic facial animation of a 3D avatar. The exact head pose estimation and facial expression tracking are critical issues that must be solved when developing vision-based computer animation. In this paper, we deal with these two problems. The proposed approach consists of two phases: dynamic head pose estimation and facial expression cloning. The dynamic head pose estimation can robustly estimate a 3D head pose from input video images. Given an initial reference template of a face image and the corresponding 3D head pose, the full head motion is recovered by projecting a cylindrical head model onto the face image. It is possible to recover the head pose regardless of light variations and self-occlusion by updating the template dynamically. In the phase of synthesizing the facial expression, the variations of the major facial feature points of the face images are tracked by using optical flow and the variations are retargeted to the 3D face model. At the same time, we exploit the RBF (Radial Basis Function) to deform the local area of the face model around the major feature points. Consequently, facial expression synthesis is done by directly tracking the variations of the major feature points and indirectly estimating the variations of the regional feature points. From the experiments, we can prove that the proposed vision-based facial expression cloning method automatically estimates the 3D head pose and produces realistic 3D facial expressions in real time.

얼굴 모션 추정과 표정 복제에 의한 3차원 얼굴 애니메이션

본 논문에서는 강건한 얼굴 포즈 추정과 실시간 표정제어가 가능한 비전 기반 3차원 얼굴 모델의 자동 표정 생성 방법 및 시스템을 제안한다. 기존의 비전 기반 3차원 얼굴 애니메이션에 관한 연구는 얼굴의 움직임을 나타내는 모션 추정을 반영하지 못하고 얼굴 표정 생성에 초점을 맞추고 있다. 그러나, 얼굴 포즈를 정확히 추정하여 반영하는 작업은 현실감 있는 얼굴 애니메이션을 위해서 중요한 이슈로 인식되고 있다. 본 연구 에서는 얼굴 포즈추정과 얼굴 표정제어가 동시에 가능한 통합 애니메이션 시스템을 제안 하였다. 제안된 얼굴 모델의 표정 생성 시스템은 크게 얼굴 검출, 얼굴 모션 추정, 표정 제어로 구성되어 있다. 얼굴 검출은 비모수적 HT 컬러 모델과 템플릿 매칭을 통해 수행된다. 검출된 얼굴 영역으로부터 얼굴 모션 추정과 얼굴 표정 제어를 수행한다. 얼굴 모션 추정을 위하여 3차원 실린더 모델을 검출된 얼굴 영역에 투영하고 광류(optical flow) 알고리즘을 이용하여 얼굴의 모션을 추정하며 추정된 결과를 3차원 얼굴 모델에 적용한다. 얼굴 모델의 표정을 생성하기 위해 특징점 기반의 얼굴 모델 표정 생성 방법을 적용한다. 얼굴의 구조적 정보와 템플릿 매칭을 이용하여 주요 얼굴 특징점을 검출하며 광류 알고리즘에 의하여 특징점을 추적한다. 추적된 특징점의 위치는 얼굴의 모션 정보와 표정 정보의 조합으로 이루어져있기 때문에 기하학적 변환을 이용하여 얼굴의 방향이 정면이었을 경우의 특징점의 변위인 애니메이션 매개변수(parameters)를 계산한다. 결국 얼굴 표정 복제는 두 개의 정합과정을 통해 수행된다. 애니메이션 매개변수 3차원 얼굴 모델의 주요 특징점(제어점)의 이동은 획득된 애니메이션 매개변수를 적용하여 수행하며, 정점 주위의 부가적 정점의 위치는 RBF(Radial Basis Function) 보간법을 통해 변형한다. 실험결과 본 논문에서 제안된 비전기반 애니메이션 시스템은 비디오 영상으로부터 강건한 얼굴 포즈 추정과 얼굴의 표정변화를 잘 반영하여 현실감 있는 애니메이션을 생성함을 입증할 수 있었다. This paper presents vision-based 3D facial expression animation technique and system which provide the robust 3D head pose estimation and real-time facial expression control. Many researches of 3D face animation have been done for the facial expression control itself rather than focusing on 3D head motion tracking. However, the head motion tracking is one of critical issues to be solved for developing realistic facial animation. In this research, we developed an integrated animation system that includes 3D head motion tracking and facial expression control at the same time. The proposed system consists of three major phases: face detection, 3D head motion tracking, and facial expression control. For face detection, with the non-parametric HT skin color model and template matching, we can detect the facial region efficiently from video frame. For 3D head motion tracking, we exploit the cylindrical head model that is projected to the initial head motion template. Given an initial reference template of the face image and the corresponding head motion, the cylindrical head model is created and the foil head motion is traced based on the optical flow method. For the facial expression cloning we utilize the feature-based method, The major facial feature points are detected by the geometry of information of the face with template matching and traced by optical flow. Since the locations of varying feature points are composed of head motion and facial expression information, the animation parameters which describe the variation of the facial features are acquired from geometrically transformed frontal head pose image. Finally, the facial expression cloning is done by two fitting process. The control points of the 3D model are varied applying the animation parameters to the face model, and the non-feature points around the control points are changed by use of Radial Basis Function(RBF). From the experiment, we can prove that the developed vision-based animation system can create realistic facial animation with robust head pose estimation and facial variation from input video image.

Real-time facial expression recognition based on pixel-pattern-based texture feature

Proposed is a pixel-pattern-based texture feature (PPBTF) for realtime facial expression recognition. Grey-scale images are transformed into pattern maps where edges and lines are used for characterising facial texture. Based on the pattern map, a feature vector is constructed. Adaboost and a support vector machine (SVM) are adopted. Experiments on the Cohn-Kanade database illustrate that the PPBTF is effective and efficient for facial expression recognition.

Internet communication using real-time facial expression analysis and synthesis

In this paper, the authors have developed a system that animates 3D facial agents based on real-time facial expression analysis techniques and research on synthesizing facial expressions and text-to-speech capabilities. This system combines visual, auditory, and primary interfaces to communicate one coherent multimodal chat experience. Users can represent themselves using agents they select from a group that we have predefined. When a user shows a particular expression while typing a text, the 3D agent at the receiving end speaks the message aloud while it replays the recognized facial expression sequences and also augments the synthesized voice with appropriate emotional content. Because the visual data exchange is based on the MPEG-4 high-level Facial Animation Parameter for facial expressions (FAP 2), rather than real-time video, the method requires very low bandwidth.

Communication over the Internet using a 3D Agent with Real-time Facial Expression Analysis and Synthesis

Communication over the Internet using a 3D Agent with Real-time Facial Expression Analysis and Synthesis

Modeling and animation of individualized faces for 3D facial expression synthesis

AbstractThis article presents new methods for realistic and fast 3D facial expression synthesis based on an individualized face model constructed from the anatomical perspective. The face model has a 3D structure of anatomically motivated facial skin, muscles, and skull. We start from a highly accurate facial mesh reconstructed from the individual face measurements. By taking as input a collection of reflectance images covering the face, the view‐based texture blending method automatically generates a comprehensive texture map for photorealistic rendering. Based on the reconstructed facial mesh, a deformable multilayer skin model is developed to simulate the dynamic behavior of the skin by taking into account the nonlinear stress‐strain relationship and incompressibility of the skin. We develop a muscle modeling process that includes three kinds of force‐based facial muscle model to simulate facial muscle contraction. The 3D face model incorporates a skull structure which extends the scope of facial motion, facilitates facial muscle construction and constrains skin deformation. Based on the facial action coding system, various facial expressions are synthesized by the contraction of a set of facial muscles. Lagrangian mechanics governs the dynamics, dictating the deformation of facial skin in response to the contraction of muscles. For computational efficiency, we devise an adaptive computation algorithm for the numerical simulation. It uses either a semi‐implicit integration scheme or a quasi‐static solver to compute the relaxation of the facial mesh by traversing the designed data structures in a breadth‐first order. The resulting system enables us to construct realistic face models of living humans and to synthesize facial expressions in real time. © 2003 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13: 42–64, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10041