MAHNOB-HCI Database Research Articles

Multimodal insights into granger causality connectivity: Integrating physiological signals and gated eye-tracking data for emotion recognition using convolutional neural network

This study introduces a groundbreaking method to enhance the accuracy and reliability of emotion recognition systems by combining electrocardiogram (ECG) with electroencephalogram (EEG) data, using an eye-tracking gated strategy. Initially, we propose a technique to filter out irrelevant portions of emotional data by employing pupil diameter metrics from eye-tracking data. Subsequently, we introduce an innovative approach for estimating effective connectivity to capture the dynamic interaction between the brain and the heart during emotional states of happiness and sadness. Granger causality (GC) is estimated and utilized to optimize input for a highly effective pre-trained convolutional neural network (CNN), specifically ResNet-18. To assess this methodology, we employed EEG and ECG data from the publicly available MAHNOB-HCI database, using a 5-fold cross-validation approach. Our method achieved an impressive average accuracy and area under the curve (AUC) of 91.00 % and 0.97, respectively, for GC-EEG-ECG images processed with ResNet-18. Comparative analysis with state-of-the-art studies clearly shows that augmenting ECG with EEG and refining data with an eye-tracking strategy significantly enhances emotion recognition performance across various emotions.

EMOTION RECOGNITION FROM HEART RATE VARIABILITY WITH A HYBRID SYSTEM COMBINED HIDDEN MARKOV MODEL AND POINCARE PLOT

The best emotion recognition system based on physiological signals with a simple operatory should have higher accuracy and fast response speed. This paper aims to develop an emotion recognition system using a novel hybrid system based on Hidden Markov Model and Poincare plot. For this purpose, an electrocardiogram from the MAHNOB-HCI database was used. A novel feature extraction from a hybrid system combining Hidden Markov Model and Poincare plot was presented. The authors extracted time and frequency domain features from heart rate variability, and used two central hybrid systems, the Support Vector Machine/ Hidden Markov Model and the Hidden Markov Model/ Poincare Plot. Finally, the support vector machine was used as a classifier to classify emotions into positive and negative. The proposed method showed a classification accuracy of 95.02 ± 1.97 % overall. Also, the computing time of the method is around 163 milliseconds. The key of this paper is in the use of hybrid machines to improve accuracy without high computation time. This method can be used as a real-time system due to the low computation time and can be developed in many fields, such as medical examination and security systems.

A novel non-contact heart rate measurement method based on EEMD combined with FastICA

Objective. The imaging photoplethysmography (IPPG), which is applied to heart rate measurement, is susceptible to interference caused by illumination variations during photography. To improve the accuracy, robustness, and real-time performance of IPPG, a new method combined with modified ensemble empirical mode decomposition (EEMD) and fast independent component analysis (FastICA) was proposed. Approach. On the modified EEMD, to reduce the error caused by the residual white noise of the algorithm, it was characterized by recurrently updating zero-mean white noise for ensemble averaging based on the standard deviation of the input signal. On the modified FastICA, its constructed nonlinear function was superseded by Huber’s approximation function to improve the robustness and running speed. Main results. Comparison experiments were conducted between the MAHNOB-HCI database and own collected data. In the hybrid natural light and computer screen light scenario, the mean absolute error (MAE) of heart rate amounted to 0.93 beats per minute and the correlation coefficient reached 0.85. In the experiments of MAHNOB-HCI database, the MAE amounted to 6.03 beats per minute and the correlation coefficient reached 0.75. Furthermore, the modified method decreased the running time by approximately ten times compared to the original algorithms. Significance. Various experimental results demonstrated that the proposed method significantly improves the accuracy, timeliness, and interference resistance.

CNN-LSTM for automatic emotion recognition using contactless photoplythesmographic signals

Numerous researchers in the field of automatic human emotion recognition have shown that physiological responses in humans are influenced by emotional changes. According to several previously conducted studies, the physiological data used are collected by sensors placed in the human body. In recent years, researchers have shown that the human heart rate can be estimated from human face videos. This technique is based on extracting the PPG signal by measuring the red–greenblue (RGB) color changes of the face of a person during the cardiac cycle. In this respect, a new framework based on contactless PPG signals for classifying human emotions is proposed in the present study. The experiment conducted in the present work used a widely popular emotional database, i.e. the MAHNOB-HCI database, to evaluate some PPG signal extraction methods. Regarding classification, a deep learning architecture, which combines a one-dimensional convolution neural network (1DCNN) and a long short-term memory (LSTM), after applying the normalization and signal segmentation steps, was adopted. It is important to emphasize that the method adopted here achieved a recognition rate of 73.33% and 60% for the binary classification of valence and arousal, respectively, with a signal segmentation of 4 s. It is worth recalling that the present paper aims primarily to propose a new study approach on automatic emotion recognition using deep learning techniques that are based on contactless sensing of physiological signals.© 2022 The Authors. Published by Elsevier B.V.

A Hybrid EEG-based Emotion Recognition Approach Using Wavelet Convolutional Neural Networks and Support Vector Machine.

Nowadays, deep learning and convolutional neural networks (CNNs) have become widespread tools in many biomedical engineering studies. CNN is an end-to-end tool, which makes the processing procedure integrated, but in some situations, this processing tool requires to be fused with machine learning methods to be more accurate. In this paper, a hybrid approach based on deep features extracted from wavelet CNNs (WCNNs) weighted layers and multiclass support vector machine (MSVM) was proposed to improve the recognition of emotional states from electroencephalogram (EEG) signals. First, EEG signals were preprocessed and converted to Time-Frequency (T-F) color representation or scalogram using the continuous wavelet transform (CWT) method. Then, scalograms were fed into four popular pre-trained CNNs, AlexNet, ResNet-18, VGG-19, and Inception-v3 to fine-tune them. Then, the best feature layer from each one was used as input to the MSVM method to classify four quarters of the valence-arousal model. Finally, the subject-independent leave-one-subject-out criterion was used to evaluate the proposed method on DEAP and MAHNOB-HCI databases. Results showed that extracting deep features from the earlier convolutional layer of ResNet-18 (Res2a) and classifying using the MSVM increased the average accuracy, precision, and recall by about 20% and 12% for MAHNOB-HCI and DEAP databases, respectively. Also, combining scalograms from four regions of pre-frontal, frontal, parietal, and parietal-occipital and two regions of frontal and parietal achieved the higher average accuracy of 77.47% and 87.45% for MAHNOB-HCI and DEAP databases, respectively. Combining CNN and MSVM increased the recognition of emotion from EEG signals and the results were comparable to state-of-the art studies.

Heart rate estimation via self-adaptive region selection and multiregion-fusion 1D CNN

Heart rate (HR) measurement from facial videos makes physiologic detection more convenient, which achieves much attention of many researchers. However, the existing methods are not very robust against motion disturbance caused by nonrigid movements of the face (e.g., smiling, yawning, and talking). This is due to the illumination variation on the local skin region, which results in the fluctuation of color intensity values in the region of interests. Different from previous approaches using fixed regions, we propose a self-adaptive region selection algorithm and then learn a 1D convolution neural network (CNN) by fusing these selected regions for HR estimation. Concretely, the observation blood volume pulse (BVP) matrix with sub-BVP is constructed from different separated regions and denoised by the rank minimization. To explore the sub-BVP of different regions, a clustering-based method is designed to generate the regional weights. In addition, a self-adaptive region selection is constructed to dynamically filter some useful face regions for robust HR estimation. Eventually, the regions and weights are separately embedded into a 1D CNN by a multiregion-fusion module. Experiments on the MAHNOB-HCI database show that the proposed approach is effective and reaches the best performance over five metrics.

Transformers for EEG-Based Emotion Recognition: A Hierarchical Spatial Information Learning Model

The spatial information of Electroencephalography (EEG) is essential for emotion recognition model to learn discriminative feature. The convolutional networks and recurrent networks are the conventional choices to learn the complex spatial dependencies through a number of electrodes and brain regions. However, these models have difficulty in capturing long-range dependencies due to the operations of local feature learning. To enhance EEG spatial dependencies capturing and improve the accuracy of emotion recognition, we propose a transformer- based model to hierarchically learn the discriminative spatial information from electrode level to brain-region-level. In the electrode-level spatial learning, the transformer encoders are adopted to integrate information within different brain regions. Next, in view of the different roles of brain regions in the emotion recognition, the self-attention within the transformer could emphasize the contributive brain regions. Hence, in the brain-region-level spatial learning, a transformer encoder is utilized to capture the spatial dependencies among the brain regions. Finally, to validate the effectiveness of the proposed model, the subject-independent experiments are conducted on the DEAP and MAHNOB-HCI database. The experimental results demonstrate that the proposed model achieves outstanding performance in emotion recognition with arousal and valence level. Moreover, the visualization of self-attention indicates that the proposed model could emphasize the discriminative spatial information from pre-frontal lobe, frontal lobe, temporal lobe and parietal lobe.

Recognition of emotional states using frequency effective connectivity maps through transfer learning approach from electroencephalogram signals

Convolutional Neural Networks (CNN) have been widely utilized in Emotion Recognition (ER) research due to their vast benefits. Designing specific CNN configuration and learning accurate parameters are provided through the Transfer Learning (TL) approach. A novel combined ER schema based on sophisticated Frequency Effective Connectivity (FEC) maps and transfer learned CNN models named FEC-CNN is proposed in this study. The EC measure estimates the information flow between brain channels in frequency bands regarding the volume conduction problem. Then, visualization of this information provides valuable and distinctive information for classifying emotional states. After preprocessing Electroencephalogram (EEG) signals, the Effective Connectivity maps were computed through the Partial Directed Coherence (PDC) measure from EEG channels at standard delta, theta, alpha, beta, and gamma frequency bands. Then, PDC maps were used to fine-tune six pre-trained CNN models, AlexNet, ResNet-18, DarkNet-19, ShuffleNet, Inception-v3, and Xception, and classify five emotional states toward the subject independent Leave-One-Subject-Out (LOSO) Cross-Validation criterion for MAHNOB-HCI, DEAP and DREAMER databases. The experimental results show that ResNet-18 achieved the highest average accuracy and F-score 94.27%, 94.74% for the DEAP database, and 95.25% and 95.33% for the MAHNOB-HCI database, and 96.00% and 96.77% for DREAMER database, to recognize five emotional states from PDC maps at alpha frequency bands. Comparison of the experimental results with recent researches showed the superiority of the proposed EC-based CNN model.This research introduced an interesting sight of deep learning applications in cognitive computation area by utilizing effective connectivity information flows to recognize emotional states from EEG signals.

Emotion recognition using effective connectivity and pre-trained convolutional neural networks in EEG signals.

Convolutional Neural Networks (CNN) have recently made considerable advances in the field of biomedical signal processing. These methodologies can assist in emotion recognition for affective brain computer interface. In this paper, a novel emotion recognition system based on the effective connectivity and the fine-tuned CNNs from multichannel Electroencephalogram (EEG) signal is presented. After preprocessing EEG signals, the relationships among 32 channels of EEG in the form of effective brain connectivity analysis which represents information flow between regions are computed by direct Directed Transfer Function (dDTF) method which yields a 32*32 image. Then, these constructed images from EEG signals for each subject were fed as input to four versions of pre-trained CNN models, AlexNet, ResNet-50, Inception-v3 and VGG-19 and the parameters of these models are fine-tuned, independently. The proposed deep learning architectures automatically learn patterns in the constructed image of the EEG signals in frequency bands. The efficiency of the proposed approach is evaluated on MAHNOB-HCI and DEAP databases. The experiments for classifying five emotional states show that the ResNet-50 applied on dDTF images in alpha band achieves best results due to specific architecture which captures the brain connectivity, efficiently. The accuracy and F1-score values for MAHNOB-HCI were obtained 99.41, 99.42 and for DEAP databases, 98.17, and 98.23. Newly proposed model is capable of effectively analyzing the brain function using information flow from multichannel EEG signals using effective connectivity measure of dDTF and ResNet-50.

PulseGAN: Learning to Generate Realistic Pulse Waveforms in Remote Photoplethysmography.

Remote photoplethysmography (rPPG) is a non-contact technique for measuring cardiac signals from facial videos. High-quality rPPG pulse signals are urgently demanded in many fields, such as health monitoring and emotion recognition. However, most of the existing rPPG methods can only be used to get average heart rate (HR) values due to the limitation of inaccurate pulse signals. In this paper, a new framework based on generative adversarial network, called PulseGAN, is introduced to generate realistic rPPG pulse signals through denoising the chrominance (CHROM) signals. Considering that the cardiac signal is quasi-periodic and has apparent time-frequency characteristics, the error losses defined in time and spectrum domains are both employed with the adversarial loss to enforce the model generating accurate pulse waveforms as its reference. The proposed framework is tested on three public databases. The results show that the PulseGAN framework can effectively improve the waveform quality, thereby enhancing the accuracy of HR, the interbeat interval (IBI) and the related heart rate variability (HRV) features. The proposed method significantly improves the quality of waveforms compared to the input CHROM signals, with the mean absolute error of AVNN (the average of all normal-to-normal intervals) reduced by 41.19%, 40.45%, 41.63%, and the mean absolute error of SDNN (the standard deviation of all NN intervals) reduced by 37.53%, 44.29%, 58.41%, in the cross-database test on the UBFC-RPPG, PURE, and MAHNOB-HCI databases, respectively. This framework can be easily integrated with other existing rPPG methods to further improve the quality of waveforms, thereby obtaining more reliable IBI features and extending the application scope of rPPG techniques.

NeuroSense: Short-term emotion recognition and understanding based on spiking neural network modelling of spatio-temporal EEG patterns

Emotion recognition still poses a challenge lying at the core of the rapidly growing area of affective computing and is crucial for establishing a successful human–computer interaction. Identification and understanding of emotions are achieved through various measures, such as subjective self-reports, face-tracking, voice analysis, gaze-tracking, as well as the analysis of autonomic and central neurophysiological measurements. Current approaches to emotion recognition based on electroencephalography (EEG) mostly rely on various handcrafted features extracted over relatively long time windows of EEG during participants exposure to appropriate affective stimuli. In this paper, we present a short-term emotion recognition framework based on spiking neural network (SNN) modelling of spatio-temporal EEG patterns. Our method relies on EEG signal segmentation based on detection of short-term changes in facial landmarks, and as such includes no computation of handcrafted EEG features. Differences between participants’ EEG properties are taken into account via subject-dependent spike encoding in the formulated subject-independent emotion recognition task. We test our methods on the publicly available DEAP and MAHNOB-HCI databases due to the availability of both EEG and frontal face video data. Through an exhaustive hyperparameter optimisation strategy, we show that the proposed SNN-based representation of EEG spiking patterns provides valuable information for short- term emotion recognition. The obtained accuracies are 78.97% and 79.39% in arousal classification, and 67.76% and 72.12% in valence classification, on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, through the application of a brain-inspired SNN model, this study provides novel insight and helps in the understanding of the neural mechanisms involved in emotional processing in the context of audiovisual stimuli, such as affective videos. The presented results encourage the use of the proposed EEG processing methodology as a complement to existing features and methods commonly used for EEG-based emotion recognition, especially for short-term arousal recognition.

Locally robust EEG feature selection for individual-independent emotion recognition

Brain computer interface (BCI) systems can decode brain affective activities into interpretable features and facilitate emotional human–computer interaction. However, individual differences of neurophysiological responses from BCI subjects constitute a stumbling block in individual-independent emotion recognition. In this study, we propose a new locally-robust feature selection (LRFS) method to determine generalizable features of electroencephalography (EEG) within several subsets of accessible subjects. In the LRFS framework, extracted EEG features are first modeled with probability densities. By evaluating the similarity of all density functions between each two subjects, inter-individual consistency of the EEG features is described. The derived consistency determines locally-robust EEG features, wherein importance of each feature is examined according to margin loss between emotions. To fuse selected features from multiple subsets of subjects, we employ ensemble learning principle and build an emotion classifier committee. Based on public DEAP and MAHNOB-HCI databases, individual-independent classification accuracy of the LRFS-based classifier is achieved by 0.65–0.68 (DEAP) and 0.67–0.70 (MAHNOB-HCI) for arousal and valence domains, respectively. Competitiveness of the LRFS has been validated when compared with several existing feature selection methods and emotion recognition systems.

A deep learning framework for heart rate estimation from facial videos

Accurate heart rate is vital to acquiring critical physical data of human subjects. For this reason, facial video-based heart rate estimation has recently received tremendous attention owing to its simplicity and convenience. However, its accuracy, reliability and computational complexity have yet to reach the desired standards. In this work, we have endeavored to develop a novel deep learning framework for real-time estimation of heart rates by using an RGB camera. Our approach consists of the following four steps. We begin Step 1 by detecting the face and facial landmarks in the video to identify the required facial Region of Interests (ROIs). In Step 2, we extract the sequence of the mean of the green-channeled video from the facial ROIs, and explore a three-stage sequential filtering, including illumination rectification, trend removal and signal amplification. In Step 3, the Short-Time Fourier Transform (STFT) is employed to convert the 1D filtered signal into the corresponding 2D Time-Frequency Representation (TFR) for characterizing the frequencies over short time intervals. The 2D TFR allows the formulation of the heart rate estimation as a video-based supervised learning problem, which can be solved by exploring a deep Convolutional Neural Network (CNN), as is carried out in Step 4. Our approach is one of the pioneering work that proposes a deep learning framework with TFRs as input for solving the heart rate estimation from facial videos. In addition, we have developed a heart rate database, named the Pulse From Face (PFF), and used it along with the existing PURE (PUlse RatE) database [1] to train the CNN. The PFF database is released for research purpose with this paper. We have evaluated the proposed framework on the MAHNOB-HCI database [2] and the VIPL-HR database [3] and compared its performance with that of other contemporary approaches to demonstrate its efficacy.

Emotion Classification Using EEG Brain Signals and the Broad Learning System

This article presents a new user-independent emotion classification method that classifies four distinct emotions using electroencephalograph (EEG) signals and the broad learning system (BLS). The public DEAP and MAHNOB-HCI databases are used. Just one EEG electrode channel is selected for the feature extraction process. Continuous wavelet transform (CWT) is then utilized to extract the proposed gray-scale image (GSI) feature which describes the EEG brain activation in both time and frequency domains. Finally, the new BLS is constructed for the emotion classification process, which successfully upgrades the efficiency of emotion classification based on EEG brain signals. The experiment results show that the proposed work produces a robust system with high accuracy of approximately 93.1% and training process time of approximately 0.7 s for the DEAP database, as well as, the high average accuracy of approximately 94.4% and training process time of approximately 0.6 s for MAHNOB-HCI database.

Emotion Recognition Based on Weighted Fusion Strategy of Multichannel Physiological Signals.

Emotion recognition is an important pattern recognition problem that has inspired researchers for several areas. Various data from humans for emotion recognition have been developed, including visual, audio, and physiological signals data. This paper proposes a decision-level weight fusion strategy for emotion recognition in multichannel physiological signals. Firstly, we selected four kinds of physiological signals, including Electroencephalography (EEG), Electrocardiogram (ECG), Respiration Amplitude (RA), and Galvanic Skin Response (GSR). And various analysis domains have been used in physiological emotion features extraction. Secondly, we adopt feedback strategy for weight definition, according to recognition rate of each emotion of each physiological signal based on Support Vector Machine (SVM) classifier independently. Finally, we introduce weight in decision level by linear fusing weight matrix with classification result of each SVM classifier. The experiments on the MAHNOB-HCI database show the highest accuracy. The results also provide evidence and suggest a way for further developing a more specialized emotion recognition system based on multichannel data using weight fusion strategy.

Novel health monitoring method using an RGB camera

In this paper we present a novel health monitoring method by estimating the heart rate and respiratory rate using an RGB camera. The heart rate and the respiratory rate are estimated from the photoplethysmography (PPG) and the respiratory motion. The method mainly operates by using the green spectrum of the RGB camera to generate a multivariate PPG signal to perform multivariate de-noising on the video signal to extract the resultant PPG signal. A periodicity based voting scheme (PVS) was used to measure the heart rate and respiratory rate from the estimated PPG signal. We evaluated our proposed method with a state of the art heart rate measuring method for two scenarios using the MAHNOB-HCI database and a self collected naturalistic environment database. The methods were furthermore evaluated for various scenarios at naturalistic environments such as a motion variance session and a skin tone variance session. Our proposed method operated robustly during the experiments and outperformed the state of the art heart rate measuring methods by compensating the effects of the naturalistic environment.

Video-Based Heartbeat Rate Measuring Method Using Ballistocardiography

Video-based heartbeat rate measurement is a rapidly growing application in remote health monitoring. Video-based heartbeat rate measuring methods operate mainly by estimating photoplethysmography or ballistocardiography signals. These methods operate by estimating the microscopic color change in the face or by estimating the microscopic rigid motion of the head/ facial skin. However, the robustness to motion artifacts caused by illumination variance and motion variance of the subject poses main challenge. We present a video-based heartbeat rate measuring framework to overcome these problems by using the principle of ballistocardiography. In this paper, we proposed a ballistocardiography model based on Newtons third law of force and dynamics of harmonic oscillation. We formulate a framework based on the ballistocardiography model to measure the rigid involuntary head motion caused by the ejection of the blood from the heart. Our proposed framework operates by estimating the motion of multivariate feature points to estimate the heartbeat rate autonomously. We evaluated our proposed framework along with existing video-based heartbeat rate measuring methods with three databases, namely; MAHNOB HCI database, human-computer interaction database, and driver health monitoring database. Our proposed framework outperformed existing methods by reporting a low mean error rate of 4.34 bpm with a standard deviation of 3.14 bpm, root mean square error of 5.29 with a high Pearson correlation coefficient of 0.91. The proposed method also operated robustly in the human-computer interaction database and driver health monitoring database by overcoming the issues related to illumination and motion variance.

Bivariate empirical mode decomposition for ECG-based biometric identification with emotional data.

Emotions modulate ECG signals such that they might affect ECG-based biometric identification in real life application. It motivated in finding good feature extraction methods where the emotional state of the subjects has minimum impacts. This paper evaluates feature extraction based on bivariate empirical mode decomposition (BEMD) for biometric identification when emotion is considered. Using the ECG signal from the Mahnob-HCI database for affect recognition, the features were statistical distributions of dominant frequency after applying BEMD analysis to ECG signals. The achieved accuracy was 99.5% with high consistency using kNN classifier in 10-fold cross validation to identify 26 subjects when the emotional states of the subjects were ignored. When the emotional states of the subject were considered, the proposed method also delivered high accuracy, around 99.4%. We concluded that the proposed method offers emotion-independent features for ECG-based biometric identification. The proposed method needs more evaluation related to testing with other classifier and variation in ECG signals, e.g. normal ECG vs. ECG with arrhythmias, ECG from various ages, and ECG from other affective databases.

Multi-modal emotion analysis from facial expressions and electroencephalogram

Automatic analysis of human spontaneous behavior has attracted increasing attention in recent years from researchers in computer vision. This paper proposes an approach for multi-modal video-induced emotion recognition, based on facial expression and electroencephalogram (EEG) technologies. Spontaneous facial expression is utilized as an external channel. A new feature, formed by percentage of nine facial expressions, is proposed for analyzing the valence and arousal classes. Furthermore, EEG is used as an internal channel supplementing facial expressions for more reliable emotion recognition. Discriminative spectral power and spectral power difference features are exploited for EEG analysis. Finally, these two channels are fused on feature-level and decision-level for multi-modal emotion recognition. Experiments are conducted on MAHNOB-HCI database, including 522 spontaneous facial expression videos and EEG signals from 27 participants. Moreover, human perception in emotion recognition compared to the proposed approach is tested with 10 volunteers. The experimental results and comparisons with the average human performance show the effectiveness of the proposed multi-modal approach.

Heartbeat Rate Measurement from Facial Video

Heartbeat rate (HR) reveals a person's health. This article presents an effective system for measuring HR from facial videos acquired in a more realistic environment than current systems' testing environments. The proposed method uses a facial feature point-tracking method that combines a good feature to track method with a supervised descent method to overcome the limitations of currently available facial video-based HR measuring systems, including unrealistic restriction of the subject's movement and artificial lighting during data capture. A face quality assessment system is also incorporated to automatically discard low-quality faces that occur in a realistic video sequence to reduce erroneous results. The proposed method was comprehensively tested on the publicly available MAHNOB-HCI database and the authors' local dataset. Experimental results show that the proposed system outperforms existing video-based systems for HR measurement.