Facial Expressions Of Subjects Research Articles

Motion-Robust Multimodal Heart Rate Estimation Using BCG Fused Remote-PPG With Deep Facial ROI Tracker and Pose Constrained Kalman Filter

The heart rate (HR) signal is so weak in remote photoplethysmography (rPPG) and ballistocardiogram (BCG) that HR estimation is very sensitive to face and body motion disturbance caused by spontaneous head and body movements as well as facial expressions of subjects in conversation. This article proposed a novel multimodal quasi-contactless HR sensor to ensure the robustness and accuracy of HR estimation under extreme facial poses, large-motion disturbances, and multiple faces in a video for computer-aided police interrogation. Specifically, we propose a novel landmark-based approach for a deep facial region of interest (ROI) tracker and face pose constrained Kalman filter to continuously and robustly track target facial ROIs for estimating HR from face and head motion disturbances in rPPG. This motion-disturbed rPPG signal is further fused with a minimally disturbed BCG signal by the face and head movements via a bank of notch filters with a recursive weighting scheme to obtain the dominant HR frequency for final accurate HR estimation. To facilitate reproducible HR estimation research, we synchronously acquire and publicly share a multimodal data set that contains 20 sets of ECG and BCG signals as well as uncompressed, rPPG-dedicated videos from ten subjects in a stable state and large-motion state (MS) without and with large face and body movements in a sitting position. We demonstrate through experimental comparisons that the proposed multimodal HR sensor is more robust and accurate than the state-of-the-art single-modal HR sensor solely with rPPG- or BCG-based methods. The mean absolute error (MAE) of HR estimation is 7.13 BPM lower than the BCG algorithm and 3.12 BPM lower than the model-based plane-orthogonal-to-skin (POS) algorithm in the MS.

Motion-tolerant heart rate estimation from face videos using derivative filter

Imaging photoplethysmography (IPPG) technique allows us to extract blood volume pulse (BVP) signals from face videos for measuring heart rate (HR), which is useful in applications such as neonatal monitoring, telemedicine and affective computing. Because the BVP signal is small, the HR estimation results are sensitive to face motion disturbance caused by spontaneous head movements and facial expressions of subjects. In this paper, we design a novel filtering method for refining the RGB signals with motion artifacts. Based on the observation that subtle color changes of face skin are smoother than large face motions at temporal scale, we use the three-order derivative of Gaussian filter to select subtle color changes under large motions. Our method is validated on both our self-collected dataset and public dataset MAHNOB-HCI containing face videos with head movements and facial expressions. By employing the proposed filtering method to pre-process the RGB signals before BVP signal extraction, a range of IPPG methods are improved to generate robust HR estimation results under realistic situations.

Subject Know Thyself? Comparing Self-Reported and Observed Emotions and Their Influence on Political Attitudes

Do subjects’ self-reports of emotion correspond to observable evidence of emotion when subjects are exposed to complex emotional stimuli such as video news reports? We explore the answer to this question with a pilot study that uses software that machine codes facial expressions of subjects to detect discrete emotions. Subjects were randomized to receive a positive, situationally-framed news story about drone warfare or a negative episodically-framed news story about drone warfare followed by a series of standard emotion self-report questions. The data produced by automated facial action coding permits a set of nuanced tests to link the ordering complex stimuli to self-reports of emotion and to subsequent opinions about drone warfare. We find evidence that self-reports and observed emotions are weakly related and observed emotion predicts attitudes about drone warfare independent of self-reports. This suggests that self-reports, while useful, do not capture the full extent of emotional experiences that shape political attitudes.

Towards robust automatic affective classification of images using facial expressions for practical applications

Affect is an important feature of multimedia content and conveys valuable information for multimedia indexing and retrieval. Most existing studies for affective content analysis are limited to low-level features or mid-level representations, and are generally criticized for their incapacity to address the gap between low-level features and high-level human affective perception. The facial expressions of subjects in images carry important semantic information that can substantially influence human affective perception, but have been seldom investigated for affective classification of facial images towards practical applications. This paper presents an automatic image emotion detector (IED) for affective classification of practical (or non-laboratory) data using facial expressions, where a lot of real-world challenges are present, including pose, illumination, and size variations etc. The proposed method is novel, with its framework designed specifically to overcome these challenges using multi-view versions of face and fiducial point detectors, and a combination of point-based texture and geometry. Performance comparisons of several key parameters of relevant algorithms are conducted to explore the optimum parameters for high accuracy and fast computation speed. A comprehensive set of experiments with existing and new datasets, shows that the method is effective despite pose variations, fast, and appropriate for large-scale data, and as accurate as the method with state-of-the-art performance on laboratory-based data. The proposed method was also applied to affective classification of images from the British Broadcast Corporation (BBC) in a task typical for a practical application providing some valuable insights.

QUANTIFYING ATYPICALITY IN AFFECTIVE FACIAL EXPRESSIONS OF CHILDREN WITH AUTISM SPECTRUM DISORDERS.

We focus on the analysis, quantification and visualization of atypicality in affective facial expressions of children with High Functioning Autism (HFA). We examine facial Motion Capture data from typically developing (TD) children and children with HFA, using various statistical methods, including Functional Data Analysis, in order to quantify atypical expression characteristics and uncover patterns of expression evolution in the two populations. Our results show that children with HFA display higher asynchrony of motion between facial regions, more rough facial and head motion, and a larger range of facial region motion. Overall, subjects with HFA consistently display a wider variability in the expressive facial gestures that they employ. Our analysis demonstrates the utility of computational approaches for understanding behavioral data and brings new insights into the autism domain regarding the atypicality that is often associated with facial expressions of subjects with HFA.

Humor and Laughter in Patients with Cerebellar Degeneration

Humor is a complex behavior which includes cognitive, affective and motor responses. Based on observations of affective changes in patients with cerebellar lesions, the cerebellum may support cerebral and brainstem areas involved in understanding and appreciation of humorous stimuli and expression of laughter. The aim of the present study was to examine if humor appreciation, perception of humorous stimuli, and the succeeding facial reaction differ between patients with cerebellar degeneration and healthy controls. Twenty-three adults with pure cerebellar degeneration were compared with 23 age-, gender-, and education-matched healthy control subjects. No significant difference in humor appreciation and perception of humorous stimuli could be found between groups using the 3 Witz-Dimensionen Test, a validated test asking for funniness and aversiveness of jokes and cartoons. Furthermore, while observing jokes, humorous cartoons, and video sketches, facial expressions of subjects were videotaped and afterwards analysed using the Facial Action Coding System. Using depression as a covariate, the number, and to a lesser degree, the duration of facial expressions during laughter were reduced in cerebellar patients compared to healthy controls. In sum, appreciation of humor appears to be largely preserved in patients with chronic cerebellar degeneration. Cerebellar circuits may contribute to the expression of laughter. Findings add to the literature that non-motor disorders in patients with chronic cerebellar disease are generally mild, but do not exclude that more marked disorders may show up in acute cerebellar disease and/or in more specific tests of humor appreciation.

Inscrutable games? Facial expressions predict economic behavior

Neuroscientific and behavioral evidence shows that when subjects are engaged in simple economic games, they pay attention to the face of their opponents. Is this a good idea? Does the face of a decision-maker contain information about his strategy space? We tested this hypothesis by modeling facial expressions of subjects playing the Ultimatum Game. We recorded videos of 60 participants, and automatically extracted time-series of facial actions (12 action units [1], shown in Fig. ​Fig.1A.,1A., as well as pitch, yaw, and roll of the head) using the real-time facial coding system of [2,3]. We then trained non-linear support vector machines (SVM) to predict the decision of the second player from a segment of video acquired after the offer was received and before the decision was entered (n = 376). To separate the dynamics of facial behavior into different temporal scales, the data was preprocessed with a bank of Gabor filters. With this method we achieved a between-subjects cross-validation accuracy of 0.66 (chance = 0.50) in predicting decisions. Because receiving an unfair offer in the Ultimatum Game is known to evoke a differential facial expression [4], we also trained a model which can capture non-linear relations between facial expressions, fairness and decisions. To do so, we labeled each instance as fair (offer > $3) or unfair, and then trained different classifiers to be ‘experts’ on either fair or unfair offers only. In this case, out-of-sample classification accuracy increased to 0.78. For both cases, we used a foreword selection procedure to identify the most predictive features (Fig.​(Fig.1B1B). Figure 1 A. Action units (AUs) used in the analysis (image of the face created with Artnatomy [5]). B. Frequency with which a feature is selected as covariate in a logistic classifier, using increases in area under the ROC as inclusion criterion. Abstract approaches that study social decision-making usually disregard the fact that choices are made in informationally rich environments. Instead, one important goal is to model different sources of information as well as the way in which they affect decisions. The current study suggests that one important source of information about strategic decision making behavior may be the face, since, given sensitive enough instruments, this information can be measured and quantified in real-time by a computer. This also suggests that real-time analysis of facial action codes may serve as a powerful new tool for understanding strategic decision making which can complement neuroimaging techniques such as EEG and fMRI.