Emotion Recognition from Peripheral Physiological Signals: A Systematic Review of Trends, Challenges and Opportunities

Emotions affect and determine social relationships and interactions, memory and creativity, and influence the mechanisms of rational thinking and decision making. The influence of emotion on decision making has gained attention in computer science. By detecting and recognizing emotions in an automatic way, machines endeavor to ease interaction between users and multimedia content. Automatic emotion detection and recognition can be carried out through analysis of users' various behavioural, and physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), electrodermal activity (EDA), and respiration, among others. These modalities have been extensively studied individually. However, since different individuals may experience the same emotion but express it differently, the various modalities are considered complementary, and fusion of various physiological and behavioural responses is expected to improve the quality of emotion recognition systems. Nevertheless, although representative features and their multimodal integration have been studied in affective computing research for various applications, patterns that arise from the dynamic interrelation among various modalities during emotional processes have received less attention. By summarizing each physiological signal only in a number of features, one may lose information present in the underlying dynamical co-evolution of various physiological signals. Considering all these issues, this thesis aims at detecting and recognizing emotion through brain and peripheral signals, targeting three complementary topics that have not been thoroughly explored. The first one includes emotion assessment from music video clips, the second one emotion assessment from odors, and the third one Quality of Experience (QoE) assessment from two-dimensional (2D) and three-dimensional (3D) video contents, using in all cases brain and peripheral physiological signals. Regarding emotion assessment from music video clips, subject-dependent and subject-independent analyses are carried out in this thesis, and the results reveal that although there are differentiations among the subjects' brain activation patterns, there are still common patterns across them. Moreover, the dynamical co-evolution between EEG and EDA is explored during emotional processes, and the results reveal that the coupling between EDA and EEG of the temporal lobe increases when strong emotions occur with respect to neutral ones. Finally, possible clustering patterns across subject-categories are investigated, and the results reveal that there are common characteristics across subject-categories related to their emotions. Regarding emotion assessment from odors, since the primary response to odors is related to pleasantness perception, which has not yet been thoroughly investigated, this thesis explores the way perceived odor pleasantness influences brain and periphery. In particular, two independent classifiers are trained and tested, one using EEG and the other using ECG features. The results reveal that it is possible to assess odor pleasantness perception from EEG and less accurately from ECG features, in a subject-independent framework. Also, decision fusion of the EEG and ECG classifiers is shown to discriminate odor pleasantness perception. Moreover, in order to explore the dynamical co-evolution between brain and peripheral signals, the coupling between heart rate and EEG is investigated. The results reveal that there is a significant increase in the coupling between ECG and temporal lobe EEG, when pleasant or unpleasant odors are experienced with respect to neutral ones. Enhanced QoE from multimedia contents targets to increase users' sensation of reality, in order to induce stronger emotions and render the user more involved in the experience. In this thesis, QoE is investigated in terms of four aspects, namely perceived depth, perceived overall quality, content preference, and sensation of reality. In particular, it is revealed that it is possible to recognize perceived depth, content preference, and sensation of reality from EEG signals, but not from the peripheral ones. Also, fusion between peripheral and EEG features is found to improve the performance in some cases. Finally, the left frontal cortex seems to be activated when sensation of reality is high, indicating that high sensation of reality is related to approach-related emotional processes. Although the three topics are independently explored in this thesis, their potential integration would endeavor to create immersive multimedia systems, which could adapt their properties according to users' emotions, and enhance, thus, user-experience.

The automated recognition of human emotions plays an important role in developing machines with emotional intelligence. Major research efforts are dedicated to the development of emotion recognition methods. However, most of the affective computing models are based on images, audio, videos and brain signals. Literature lacks works that focus on utilizing only peripheral signals for emotion recognition (ER), which can be ideally implemented in daily life settings. Therefore, this paper present a framework for ER on the arousal and valence space, based on using multi-modal peripheral signals. The data used in this work were collected during a debate between two people using wearable devices. The emotions of the participants were rated by multiple raters and converted into classes in correspondence to the arousal and valence space. The use of a dynamic threshold for ratings conversion was investigated. An ER model is proposed that uses a Long Short-Term Memory (LSTM)-based architecture for classification. The model uses heart rate (HR), temperature (T), and electrodermal activity (EDA) signals as its inputs with emotional cues. Additionally, a post-processing prediction mechanism is introduced to enhance the recognition performance. The model is implemented to study the use of individual and different combinations of the peripheral signals, as well as utilizing annotations from different ratings. Additionally, it is employed for classification of valence and arousal in an independent and combined fashion, under subject dependent and independent scenarios. The experimental results have justified the efficient performance of the proposed framework, achieving classification accuracy 96% and 93% for the independent and combined classification scenarios, accordingly. The comparison of the achieved performance against the baseline methods shows the superiority of the proposed framework and the ability to recognize arousal-valance levels with high accuracy from peripheral signals, in real-life scenarios.

Recent developments in wearable technology have led to increased research interest in using peripheral physiological signals for emotion recognition. The non-invasive nature of peripheral physiological signal measurement via wearables enables ecologically valid long-term monitoring. These peripheral signal measurements can be used in real-time in many ways including health and emotion classification. This paper investigates the utility of peripheral physiological signals for emotion recognition using the publicly available DEAP database. Using this database (which contains electroencephalogram (EEG) signals and peripheral signals), this paper compares eight machine learning models in the classification of valence and arousal emotion dimensions. These were applied to the peripheral physiological signals only. These models operate on three groupings of the peripheral data: (i) the raw peripheral physiological signals; (ii) individual feature sets extracted from each peripheral signal; and (iii) a fusion data set made of the combined features from the individual peripheral signals. The results indicate that support vector machine, linear discriminant analysis and logistic regression give the best recognition results on all three data groups considered. The feature fusion data set, which is made up by fusing all the features from the peripheral signals, gives the best recognition accuracy on both valence and arousal dimensions. In addition, subject dependency for emotion classification from peripheral signals is examined and significant individual variability is observed. The recognition rate varies between each participant from 10% to 87.5%.

Current multi-modal emotion recognition from physiological signals requires electroencephalogram(EEG) signals and peripheral physiological signals during both training and test. Compared with the peripheral physiological signals, it is more difficult to obtain EEG signals in our daily life. Therefore, we propose a novel approach to recognize emotions from peripheral signals by using EEG features as privileged information, which is only available during training. During training, first, peripheral physiological features and EEG features are extracted. Then, we construct a new peripheral physiological feature space using canonical correlation analysis with the help of EEG features. Finally we train a support vector machine(SVM) to map the new peripheral physiological features to the emotion labels. During test, only peripheral physiological features are used to recognize emotions from the constructed peripheral physiological feature space with the trained SVM model. The experimental results on two benchmark databases show that our proposed approach using EEG features as privileged information outperforms the method which recognizes emotions merely from the peripheral physiological signals.

Purpose The aim of this study is to investigate subject independent emotion recognition capabilities of EEG and peripheral physiological signals namely: electroocoulogram (EOG), electromyography (EMG), electrodermal activity (EDA), temperature, plethysmograph and respiration. The experiments are conducted on both modalities independently and in combination. This study arranges the physiological signals in order based on the prediction accuracy obtained on test data using time and frequency domain features. Design/methodology/approach DEAP dataset is used in this experiment. Time and frequency domain features of EEG and physiological signals are extracted, followed by correlation-based feature selection. Classifiers namely – Naïve Bayes, logistic regression, linear discriminant analysis, quadratic discriminant analysis, logit boost and stacking are trained on the selected features. Based on the performance of the classifiers on the test set, the best modality for each dimension of emotion is identified. Findings The experimental results with EEG as one modality and all physiological signals as another modality indicate that EEG signals are better at arousal prediction compared to physiological signals by 7.18%, while physiological signals are better at valence prediction compared to EEG signals by 3.51%. The valence prediction accuracy of EOG is superior to zygomaticus electromyography (zEMG) and EDA by 1.75% at the cost of higher number of electrodes. This paper concludes that valence can be measured from the eyes (EOG) while arousal can be measured from the changes in blood volume (plethysmograph). The sorted order of physiological signals based on arousal prediction accuracy is plethysmograph, EOG (hEOG + vEOG), vEOG, hEOG, zEMG, tEMG, temperature, EMG (tEMG + zEMG), respiration, EDA, while based on valence prediction accuracy the sorted order is EOG (hEOG + vEOG), EDA, zEMG, hEOG, respiration, tEMG, vEOG, EMG (tEMG + zEMG), temperature and plethysmograph. Originality/value Many of the emotion recognition studies in literature are subject dependent and the limited subject independent emotion recognition studies in the literature report an average of leave one subject out (LOSO) validation result as accuracy. The work reported in this paper sets the baseline for subject independent emotion recognition using DEAP dataset by clearly specifying the subjects used in training and test set. In addition, this work specifies the cut-off score used to classify the scale as low or high in arousal and valence dimensions. Generally, statistical features are used for emotion recognition using physiological signals as a modality, whereas in this work, time and frequency domain features of physiological signals and EEG are used. This paper concludes that valence can be identified from EOG while arousal can be predicted from plethysmograph.

Natural human-system interaction can facilitate the acceptance of technological systems. The ability of emotion recognition can hereby provide a significant contribution. Surprisingly, the field of emotion recognition is dominated by static machine learning approaches that do not account for the dynamics present in emotional processes. To overcome this limitation, we applied nonlinear autoregressive (NARX) models to predict emotion intensity from different physiological features extracted from galvanic skin response (GSR), heart rate (HR) and respiration (RSP) signals. NARX models consider the history of both the exogenous inputs (physiological signals) and the output (intensity). Emotions of different intensities were induced with images, while the physiological signals were recorded and the participants assessed their subjectively felt intensity in real-time. The intensity changes were analysed for three different emotion qualities: Happiness/Joy, Disappointment/Regret, Worry/Fear. While models were obtained for each individual, only the best set of parameters across individuals was considered for evaluation. Overall, it was found that the NARX models performed better than a sliding-window linear regression for all qualities. Furthermore, relevant features for the prediction of intensity and “ideal” delays between physiological features and the felt intensity to be captured by the model were identified. Overall, results underline the importance of considering dynamics in emotion recognition and prediction tasks.KeywordsEmotion recognitionEmotion dynamicsAppraisal modelsElectrophysiological signalsEmotion intensity

Electroencephalography (EEG) signals provide a promising and involuntary reflection of brain activity related to emotional states, offering significant advantages over behavioral cues such as facial expressions. However, EEG signals are often noisy, affected by artifacts, and vary across individuals, complicating emotion recognition. While multimodal approaches have used peripheral physiological signals (PPS) such as galvanic skin response to complement EEG, they often overlook the dynamic synchronization and consistent semantics between the modalities. Additionally, the temporal dynamics of emotional fluctuations across different time resolutions in PPS remain underexplored. To address these challenges, we propose PhysioSync, a novel pretraining framework leveraging temporal and cross-modal contrastive learning (CM-CL), inspired by physiological synchronization phenomena. PhysioSync incorporates cross-modal consistency alignment (CM-CA) to model dynamic relationships between EEG and complementary PPS, enabling emotion-related synchronizations across modalities. Besides, it introduces long- and short-term temporal contrastive learning (LS-TCL) to capture emotional synchronization at different temporal resolutions within modalities. After pretraining, cross-resolution and cross-modal features are hierarchically fused and fine-tuned to enhance emotion recognition. Experiments on DEAP and DREAMER datasets demonstrate PhysioSync’s advanced performance under unimodal and cross-modal conditions, highlighting its effectiveness for EEG-centered emotion recognition.

Emotion recognition plays a particularly important role in the field of artificial intelligence. However, the emotional recognition of electroencephalogram (EEG) in the past was only a unimodal or a bimodal based on EEG. This paper aims to use deep learning to perform emotional recognition based on the multimodal with valence-arousal dimension of EEG, peripheral physiological signals, and facial expressions. The experiment uses the complete data of 18 experimenters in the Database for Emotion Analysis Using Physiological Signals (DEAP) to classify the EEG, peripheral physiological signals and facial expression video in unimodal and multimodal fusion. The experiment demonstrates that Multimodal fusion's accuracy is excelled that in unimodal and bimodal fusion. The multimodal compensates for the defects of unimodal and bimodal information sources.

Emotion recognition based on multiple physiological signals

Emotion recognition becomes an investigated topic in affective computing for several applications. The presented paper aims to recognize human emotions using peripheral physiological signals as well as electrocardiogram (ECG), galvanic skin response (GSR), Skin Temperature (Temp) and respiration volume (RV). To achieve this purpose, we develop our work with the multimodal database MAHNOB-HCI. The emotional responses of twenty four participants to twenty affective stimuli videos are classified into two precise areas in valence-arousal emotional space. Using the support vector machine (SVM) as a classifier, the results, over the mentioned signals, show that the ECG and RV signals are the most relevant for emotion recognition issue. Moreover, our obtained accuracies are promising compared to related work.

In recent years, emotion recognition has received increasing attention as it plays an essential role in human-computer interaction systems. This paper proposes a four-class multimodal approach for emotion recognition based on peripheral physiological signals that uniquely combines a Continuous Wavelet Transform (CWT) for feature extraction, an overlapping sliding window approach to generate more data samples and a Convolutional Neural Network (CNN) model for classification. The proposed model processes multiple signal types such as Galvanic Skin Response (GSR), respiration patterns, and blood volume pressure. Achieved results indicate an accuracy of 84.2%, which outperforms state-of-the-art models on four-class classification despite of being only based on peripheral signals.KeywordsEmotion recognitionConvolutional neural networkDeep learningPhysiological signalsWavelet transform

Emotion recognition in EEG signals using deep learning methods: A review

The interest in development of methods and tools for recognizing human emotions has increased continuously. Using physiological information, especially the peripheral physiological signals, to identify emotions is an important direction for this area. This paper proposes an approach for emotion recognition based on energy-related features extracted from peripheral physiological signals. Three emotions: calm, happiness and fear, were elicited in 54 volunteers using video clips while three peripheral physiological signals were recorded: Electrocardiography (ECG), Photoplethysmography (PPG) and Respiration. Given that energy-related features of physiological signals are closely related to autonomic nervous systems activities, nine energy-related features were extracted from the recorded physiological signals. To find the optimal feature subset to represent the target emotions, the correlation between features and emotion state, as well as the discrimination ability of feature for emotion recognition were both analyzed. Four optimal features were then selected for further classification. Moreover, models based on Decision Tree (DT) were built to evaluate the performance of these features for purpose of recognition of emotion states of calm, happiness, and fear. The results show that the DT models based on these four optimal features could distinguish fear from calm (AUC=0.879, Accuracy=87.8%), happiness from calm (AUC=0.915, Accuracy=91.8%), and fear from happiness (AUC=0.822, Accuracy=81.8%), with a global recognition accuracy of 70.8%. These results indicate that energy-related features of peripheral physiological signals can reliably identify emotions, especially intense emotions.

This research aims to present a method for emotion recognition using the K-Emocon dataset (Park et al. in Sci Data 7(1):1–16 [8]) for use in the healthcare sector as well as to enhance computer–human interaction. In the following work, we use peripheral physiological signals to recognize emotion using classifier models with multidimensional emotion space models. These signals are collected using IoT-based wireless wearable devices. Emotions are measured in terms of arousal and valence by using physiological signals obtained from these devices. Several machine learning models were used for emotion recognition. Thirty-eight input features were extracted from a variety of physiological signals present in the dataset for analysis. Best accuracy achieved for valence and arousal in our experiment was 91.12% and 62.19%, respectively. This study targets recognition and classification of emotions during naturalistic conversations between people using peripheral physiological signals. It is shown that it is viable to recognize emotions using these signals.KeywordsK-EmoconEmotion recognitionMachine learningNeural networkIoTWearable sensors

The facial and physiological sensor-based emotion recognition methods are two popular methods of emotion recognition. The proposed research is the first of its kind in real-time emotion recognition that combines skin conductance signals with the visual-based facial emotion recognition (FER) method on a Raspberry Pi. This research includes stepwise documentation of method for automatic real-time face detection and FER on portable hardware. Further, the proposed work comprises experimentation related to video induction and habituation methods with FER and the galvanic skin response (GSR) method. The GSR data are recorded as skin conductance and represent the subject's behavioral changes in the form of emotional arousal and face emotion recognition on the portable device. The article provides a stepwise implementation of the following methods: (a) the skin conductance representation from the GSR sensor for arousal; (b) gathering visual inputs for identifying the human face; (c) FER from the camera module; and (d) experimentation on the proposed framework. The key feature of this article is the comprehensive documentation of stepwise implementation and experimentation, including video induction and habituation experimentation. An illuminating aspect of the proposed method is the survey of GSR trademarks and the conduct of psychological experiments. This study is useful for emotional computing systems and potential applications like lie detectors and human–machine interfaces, devices for gathering user experience input, identifying intruders, and providing portable and scalable devices for experimentation. We termed our approaches "sensovisual" (sensors + visual) and "Emosense" (emotion sensing).