Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia.

Aasim Raheel,Syed Muhammad Anwar,Muhammad Majid,Majdi Alnowami

doi:10.3390/s20144037

Aasim Raheel, Syed Muhammad Anwar + Show 2 more

Open Access

https://doi.org/10.3390/s20144037

Copy DOI

Abstract

Emotion recognition has increased the potential of affective computing by getting an instant feedback from users and thereby, have a better understanding of their behavior. Physiological sensors have been used to recognize human emotions in response to audio and video content that engages single (auditory) and multiple (two: auditory and vision) human senses, respectively. In this study, human emotions were recognized using physiological signals observed in response to tactile enhanced multimedia content that engages three (tactile, vision, and auditory) human senses. The aim was to give users an enhanced real-world sensation while engaging with multimedia content. To this end, four videos were selected and synchronized with an electric fan and a heater, based on timestamps within the scenes, to generate tactile enhanced content with cold and hot air effect respectively. Physiological signals, i.e., electroencephalography (EEG), photoplethysmography (PPG), and galvanic skin response (GSR) were recorded using commercially available sensors, while experiencing these tactile enhanced videos. The precision of the acquired physiological signals (including EEG, PPG, and GSR) is enhanced using pre-processing with a Savitzky-Golay smoothing filter. Frequency domain features (rational asymmetry, differential asymmetry, and correlation) from EEG, time domain features (variance, entropy, kurtosis, and skewness) from GSR, heart rate and heart rate variability from PPG data are extracted. The K nearest neighbor classifier is applied to the extracted features to classify four (happy, relaxed, angry, and sad) emotions. Our experimental results show that among individual modalities, PPG-based features gives the highest accuracy of as compared to EEG- and GSR-based features. The fusion of EEG, GSR, and PPG features further improved the classification accuracy to (for four emotions) when interacting with tactile enhanced multimedia.

Highlights

Human senses are physiological responses and play a vital role in perception
tactile enhanced multimedia (TEM) clips were created which would simultaneously entice three human senses including tactile, auditory, and vision. This could improve the emotional experience of a user, which was demonstrated by recognizing emotions using physiological signals
EEG, galvanic skin response (GSR), and PPG data were acquired in response to these TEM clips

Summary

Introduction

Human senses are physiological responses and play a vital role in perception. Humans perceive their surrounding environment using multiple senses, i.e., vision, auditory, gustatory, olfactory, and tactile (touch). The sensing organs transmit information to the human brain, which helps in perceiving the surrounding environment. Traditional multimedia engages only two human senses, i.e., auditory and vision. Whereas the human experience of viewing multimedia content can be enhanced by engaging more than two human senses simultaneously.

Results

Discussion

Conclusion