Abstract
Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers.
Highlights
Past research aimed at emotion detection/classification has utilized electroencephalogram (EEG) and/or facial expressions[1,2,3]
Past research in affective computing has mostly focused on emotion classification using various Audio-visual or bio-sensing modalities[1,2]
A severe limitation of such research has been its inability to boost classification accuracy to human-like levels. Another limitation of previous research has been its inability to demonstrate which cues related to Facial Expressions or EEG are most correlated with human emotions
Summary
Past research aimed at emotion detection/classification has utilized electroencephalogram (EEG) and/or facial expressions[1,2,3]. As a recent detailed survey about recognizing emotions using EEG demonstrates[5], even after the introduction of multi-modal affective datasets[1,2,6], the research has not translated from an emotion detection/classification problem to assess the correlation between the Audio-visual cues from the multimedia content and audience’s associated response through changes in the EEG. We perform the analysis on the model to provide insights into the regions from the face and brain (i.e. the audience’s physiology) that are most correlated with emotions We perform such analysis to detect what type of Audio-visual cues are most effective in evoking changes in the human physiology
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
