Abstract
Recent progress in Affective Computing (AC) has enabled integration of physiological cues and spontaneous expressions to reveal a subject’s emotional state. Due to the lack of an effective technique for evaluating multimodal correlations, experience and intuition play a main role in present AC studies when fusing affective cues or modalities, resulting in unexpected outcomes. This study seeks to demonstrate a dynamic correlation between two such affective cues, physiological changes and spontaneous expressions, which were obtained by a combination of stereo vision based tracking and imaging photoplethysmography (iPPG), with a designed protocol involving 20 healthy subjects. The two cues obtained were sampled into a Statistical Association Space (SAS) to evaluate their dynamic correlation. It is found that the probability densities in the SAS increase as the peaks in two cues are approached. Also the complex form of the high probability density region in the SAS suggests a nonlinear correlation between two cues. Finally the cumulative distribution on the zero time-difference surface is found to be small (<0.047) demonstrating a lack of simultaneity. These results show that the two cues have a close interrelation, that is both asynchronous and nonlinear, in which a peak of one cue heralds a peak in the other.
Highlights
Recent progress in Affective Computing (AC) has enabled integration of physiological cues and spontaneous expressions to reveal a subject’s emotional state
The experimental setup for the current study is shown in Expression Energy (EE) measurements were obtained from the relative displacement of detectable landmarks
To validate the heart rate (HR) measurement, a Bland-Altman analysis was performed on three subjects to assess the agreement between HR measurements obtained by imaging photoplethysmography (iPPG) and ECG
Summary
Recent progress in Affective Computing (AC) has enabled integration of physiological cues and spontaneous expressions to reveal a subject’s emotional state. This study seeks to demonstrate a dynamic correlation between two such affective cues, physiological changes and spontaneous expressions, which were obtained by a combination of stereo vision based tracking and imaging photoplethysmography (iPPG), with a designed protocol involving 20 healthy subjects. An affect-sensitive learning environment able to recognize and respond to frustration is expected to increase the motivation for study, and improve learning compared to an affect-insensitive environment Inspired by this belief, AC research has endeavored to narrow the communicative gap between the highly emotional human and an emotionally challenged computer and a number of affect-sensitive systems have been developed in several domains, including gaming, mental health, and learning technologies[2,3,4,5,6]. Due to the challenge of fusing the heterogeneous information, such a multimodal paradigm is widely advocated, but rarely implemented
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
