Multimodal Physiological Signals Research Articles

Impact of Situation Awareness Variations on Multimodal Physiological Responses in High-Speed Train Driving.

In safety-critical environments, human error is a leading cause of accidents, with the loss of situation awareness (SA) being a key contributing factor. Accurate SA assessment is essential for minimizing such risks and ensuring operational safety. Traditional SA measurement methods have limitations in dynamic real-world settings, while physiological signals, particularly EEG, offer a non-invasive, real-time alternative for continuous SA monitoring. However, the reliability of SA measurement based on physiological signals depends on the accuracy of SA labeling. This study aims to design an effective SA measurement paradigm specific to high-speed train driving, investigate more accurate physiological signal-based SA labeling methods, and explore the relationships between SA levels and key physiological metrics based on the developed framework. This study recruited 19 male high-speed train driver trainees and developed an SA measurement paradigm specific to high-speed train driving. A method combining subjective SA ratings and task performance was introduced to generate accurate SA labels. The results of statistical analysis confirmed the effectiveness of this paradigm in inducing SA level changes, revealing significant relationships between SA levels and key physiological metrics, including eye movement patterns, ECG features (e.g., heart rate variability), and EEG power spectral density across theta, alpha, and beta bands. This study supports the use of multimodal physiological signals for SA assessment and provides a theoretical foundation for future applications of SA monitoring in railway operations, contributing to enhanced operational safety.

MCNN-CMCA: A multiscale convolutional neural networks with cross-modal channel attention for physiological signal-based mental state recognition

Human mental state recognition (MSR) has significant implications for human-machine interactions. Although mental state recognition models based on single-modality signals, such as electroencephalogram (EEG) or peripheral physiological signals (PPS), have achieved encouraging progress, methods leveraging multimodal physiological signals still need to be explored. In this study, we present MCNN-CMCA, a generic model that employs multiscale convolutional neural networks (CNNs) with cross-modal channel attention to realize physiological signals-based MSR. Specifically, we first design an innovative cross-modal channel attention mechanism that adaptively adjusting the weights of each signal channel, effectively learning both intra-modality and inter-modality correlation and expanding the channel information to the depth dimension. Additionally, the study utilizes multiscale temporal CNNs for obtaining short-term and long-term time-frequency features across different modalities. Finally, the multimodal fusion module integrates the representations of all physiological signals and the classification layer implements sparse connections by setting the mask weights to 0. We evaluate the proposed method on the SEED-VIG, DEAP, and self-made datasets, achieving superior results compared to existing state-of-the-art methods. Furthermore, we conduct ablation studies to demonstrate the effectiveness of each component in the MCNN-CMCA and show the use of multimodal physiological signals outperforms single-modality signals.

Emotion recognition based on a limited number of multimodal physiological signals channels

Emotion constitutes a higher-order cognitive process, moreover, physiological signals can offer a more objective and realistically reflection of human emotional state. Electroencephalograph (EEG) signals, controlled by the central nervous system, are highly sensitive to emotional state fluctuations. Current research primarily focuses on recognizing emotions through feature extraction from multi-channel EEG signals. However, the acquisition of multi-channel EEG signals is hindered by difficulties acquiring from hair occlusion. The objective of this study is to construct a real-time physiological-emotion recognition model based on 2-channel EEG signals from the forehead without hair occlusion combined with 3-channel peripheral physiological (PPS) signals, employing signal preprocessing, feature extraction, feature normalization, and the ensemble learning algorithm Light Gradient Boosting Machine (LightGBM). In this paper, thirty participants sequentially viewed fourteen virtual reality (VR) scenes, rated for valence and arousal using the Self-Assessment Manikin (SAM) scale after each scene. The 2-channel EEG signals and 3-channel PPS signals were recorded synchronously from forehead by VR Head-Mounted Displays (HMD) Bio Pad developed in our research center. Our proposed model achieved median accuracy of 89.68% for valence, 90.11% for arousal in binary classification, and 84.93% for four classification of emotion recognition based on the collected samples. Furthermore, verification result of the proposed method on a public database (DEAP) achieved accuracy of 84.03%, 84.37%, and 72.23% on valence, arousal, and four classification, respectively. The proposed physiological-emotion recognition model, utilizing a limited number of multimodal physiological signals channels from wearable devices, demonstrates higher accuracy compared with the results in some literature, suggesting its potential for real-time implementation due to limited signal channels and low runtime.

A hybrid transposed attention based deep learning model for wearable and explainable stress recognition

Stress is a prevalent issue in modern society which affects a large percentage of the population. Stress has negative effects on human daily lives such as reduced memory and concentration levels, forgetfulness, restlessness, and weakness. Although it is challenging to extinguish stress from human lives, monitoring and controlling its consequences can be implemented. Current methods for stress monitoring often lack interpretability of results and making it difficult to provide an interpretable system for immediate mental health counseling and intervention. Wearable devices are of growing interest due to the impact of the COVID-19 pandemic on mental health, as wearables provide monitoring of physiological parameters which helps in measuring stress levels. Here, an investigation on the feasibility of using wrist and chest-based multimodal physiological signals collected from wearables is executed for stress classification and identification of underlying modalities that impact the stress. Multimodalities from wrist and chest sensors are used to examine the three-class classification, baseline, stress and amusement. For improved and visualized effects, the multimodal signals from wearables were transformed into images. Then, a hybrid model was introduced, which comprised of attentive convolution neural network (CNN), transposed attentive CNN connections with long-short term memory (LSTM) and followed by attention layer for efficient feature extraction and improved classification accuracy. Results found that the proposed model with images provided 97 % accuracy and 96 % F1-score for chest wearable and 94 % accuracy and 93 % F1-score for wrist wearable data. Additionally, the post-hoc explainability approach (Local interpretable model agnostic applications, LIME) provided visual representations of contributing features from each signal for each class. LIME shows electrodermal activity (EDA), temperature (TEMP) and respiration (RESP) to be the significant factor in stress recognition from chest wearable and blood volume pulse (BVP) and EDA from wrist wearable. Additionally, increasing the number of features of the explainable model influences the modalities influence on the model explainability. The results establish a benchmark for explainable stress recognition using different sensor data.

Automated Assessment of Driver Distraction Using Multimodal Wearable Data and Squeeze-Excitation Networks.

Driver distraction, crucial for road safety, can benefit from multimodal physiological signals assessment. However, fusion of heterogeneous data is highly challenging. In this study, we address this challenge by exploring 1D convolution neural network (CNN) with squeeze and excitation networks (SEcNN) on multimodal data. For this, electrocardiogram (256Hz) and respiration (128Hz) are obtained from subjects (N=10) while using textile electrodes and driving in different scenarios namely normal, texting and calling. The obtained multimodal data is preprocessed and SEcNN to identify driver distraction. Experiments are performed using Leave-one-out-subject cross validation. The proposed approach is able to discriminate drivers distraction. It is observed that SEcNN yields average accuracy 57.03% and average F1 score 54.90% for shorter segments. Thus, the proposed approach using wearable shirts could be useful for non-intrusive monitoring in real world driver scenarios.

A Novel Seizure Detection Method Based on the Feature Fusion of Multimodal Physiological Signals

A Novel Seizure Detection Method Based on the Feature Fusion of Multimodal Physiological Signals

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Objective. Physiological signals based emotion recognition is a prominent research domain in the field of human-computer interaction. Previous studies predominantly focused on unimodal data, giving limited attention to the interplay among multiple modalities. Within the scope of multimodal emotion recognition, integrating the information from diverse modalities and leveraging the complementary information are the two essential issues to obtain the robust representations. Approach. Thus, we propose a intermediate fusion strategy for combining low-rank tensor fusion with the cross-modal attention to enhance the fusion of electroencephalogram, electrooculogram, electromyography, and galvanic skin response. Firstly, handcrafted features from distinct modalities are individually fed to corresponding feature extractors to obtain latent features. Subsequently, low-rank tensor is fused to integrate the information by the modality interaction representation. Finally, a cross-modal attention module is employed to explore the potential relationships between the distinct latent features and modality interaction representation, and recalibrate the weights of different modalities. And the resultant representation is adopted for emotion recognition. Main results. Furthermore, to validate the effectiveness of the proposed method, we execute subject-independent experiments within the DEAP dataset. The proposed method has achieved the accuracies of 73.82% and 74.55% for valence and arousal classification. Significance. The results of extensive experiments verify the outstanding performance of the proposed method.

A multimodal dataset for investigating working memory in presence of music: a pilot study.

Decoding an individual's hidden brain states in responses to musical stimuli under various cognitive loads can unleash the potential of developing a non-invasive closed-loop brain-machine interface (CLBMI). To perform a pilot study and investigate the brain response in the context of CLBMI, we collect multimodal physiological signals and behavioral data within the working memory experiment in the presence of personalized musical stimuli. Participants perform a working memory experiment called the n-back task in the presence of calming music and exciting music. Utilizing the skin conductance signal and behavioral data, we decode the brain's cognitive arousal and performance states, respectively. We determine the association of oxygenated hemoglobin (HbO) data with performance state. Furthermore, we evaluate the total hemoglobin (HbT) signal energy over each music session. A relatively low arousal variation was observed with respect to task difficulty, while the arousal baseline changes considerably with respect to the type of music. Overall, the performance index is enhanced within the exciting session. The highest positive correlation between the HbO concentration and performance was observed within the higher cognitive loads (3-back task) for all of the participants. Also, the HbT signal energy peak occurs within the exciting session. Findings may underline the potential of using music as an intervention to regulate the brain cognitive states. Additionally, the experiment provides a diverse array of data encompassing multiple physiological signals that can be used in the brain state decoder paradigm to shed light on the human-in-the-loop experiments and understand the network-level mechanisms of auditory stimulation.

Intelligent emotion recognition in product design using multimodal physiological signals and machine learning

Identifying emotional responses in products is essential for product design and user research. Traditional methods, such as interviews and surveys, for gathering product experience data are time-consuming and resource-intensive, and often fail to capture users’ genuine emotional intentions. This article introduces an intelligent method for accurately identifying user-product emotions using multimodal physiological signals and machine learning techniques. The study involves designing experiments with 63 representative product images, and collecting various physiological signals (eye movement, electrodermal activity, and pulse). Using the K-means algorithm, we establish efficacy-arousal assessment labels to create a product emotion dataset. Preprocessing methods, including time–frequency analysis and wavelet transformation, ensure high-quality signal analysis data. By extracting temporal, frequency, and time–frequency features from multimodal physiological signals, the Relief algorithm determines the optimal feature combination. Finally, the study evaluates the performance of five machine learning classifiers, with results indicating an 81.31% recognition accuracy for the proposed product emotion recognition framework. This framework proves effective for objectively analyzing user emotions in product experiences.

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism.

Modeling correlations between multimodal physiological signals [e.g., canonical correlation analysis (CCA)] for emotion recognition has attracted much attention. However, existing studies rarely consider the neural nature of emotional responses within physiological signals. Furthermore, during fusion space construction, the CCA method maximizes only the correlations between different modalities and neglects the discriminative information of different emotional states. Most importantly, temporal mismatches between different neural activities are often ignored; therefore, the theoretical assumptions that multimodal data should be aligned in time and space before fusion are not fulfilled. To address these issues, we propose a discriminative correlation fusion method coupled with a temporal alignment mechanism for multimodal physiological signals. We first use neural signal analysis techniques to construct neural representations of the central nervous system (CNS) and autonomic nervous system (ANS). respectively. Then, emotion class labels are introduced in CCA to obtain more discriminative fusion representations from multimodal neural responses, and the temporal alignment between the CNS and ANS is jointly optimized with a fusion procedure that applies the Bayesian algorithm. The experimental results demonstrate that our method significantly improves the emotion recognition performance. Additionally, we show that this fusion method can model the underlying mechanisms in human nervous systems during emotional responses, and our results are consistent with prior findings. This study may guide a new approach for exploring human cognitive function based on physiological signals at different time scales and promote the development of computational intelligence and harmonious human-computer interactions.

MVF-SleepNet: Multi-View Fusion Network for Sleep Stage Classification.

Sleep stage classification is of great importance in human health monitoring and disease diagnosing. Clinically, visual-inspected classifying sleep into different stages is quite time consuming and highly relies on the expertise of sleep specialists. Many automated models for sleep stage classification have been proposed in previous studies but their performances still exist a gap to the real clinical application. In this work, we propose a novel multi-view fusion network named MVF-SleepNet based on multi-modal physiological signals of electroencephalography (EEG), electrocardiography (ECG), electrooculography (EOG), and electromyography (EMG). To capture the relationship representation among multi-modal physiological signals, we construct two views of Time-frequency images (TF images) and Graph-learned graphs (GL graphs). To learn the spectral-temporal representation from sequentially timed TF images, the combination of VGG-16 and GRU networks is utilized. To learn the spatial-temporal representation from sequentially timed GL graphs, the combination of Chebyshev graph convolution and temporal convolution networks is employed. Fusing the spectral-temporal representation and spatial-temporal representation can further boost the performance of sleep stage classification. A large number of experiment results on the publicly available datasets of ISRUC-S1 and ISRUC-S3 show that the MVF-SleepNet achieves overall accuracy of 0.821, F1 score of 0.802 and Kappa of 0.768 on ISRUC-S1 dataset, and accuracy of 0.841, F1 score of 0.828 and Kappa of 0.795 on ISRUC-S3 dataset. The MVF-SleepNet achieves competitive results on both datasets of ISRUC-S1 and ISRUC-S3 for sleep stage classification compared to the state-of-the-art baselines. The source code of MVF-SleepNet is available on Github (https://github.com/YJPai65/MVF-SleepNet).

AFLEMP: Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data

Automated emotion recognition systems utilizing physiological signals are essential for affective computing and intelligent interaction. Combining the multiple physiological signals is more precise and effective in accurately assessing a person’s emotional state. These automated emotion recognition systems using conventional machine learning techniques require complete access to the physiological data for emotion state classification, compromising sensitive data privacy. Federated Learning (FL) resolves this issue by preserving the user’s privacy and sensitive physiological data while recognizing emotions. However, existing FL methods have limitations in handling data heterogeneity in the physiological data and do not measure communication efficiency and scalability. In response to these challenges, this paper proposes a unique novel framework called AFLEMP (Attention-based Federated Learning for Emotion recognition using Multi-modal Physiological data) integrating attention mechanism-based Transformer with an Artificial Neural Network (ANN) model. The framework reduces two types of data heterogeneity: (1) Variation Heterogeneity (VH) in multi-modal EEG, GSR, and ECG physiological signal data using attention mechanisms and (2) Imbalanced Data Heterogeneity (IDH) in the FL environment using scaled weighted federated averaging. This paper validates the proposed AFLEMP framework on two publicly available emotion datasets, AMIGOS and DREAMER, achieving an average accuracy of 88.30% and 84.10%, respectively. The proposed AFLEMP framework proves robust, scalable, and efficient in communication. AFLEMP is the first FL framework to propose for emotion recognition using multi-modal physiological signals while reducing data heterogeneity and outperforming existing FL methods.

Comparing multimodal physiological responses to social and physical pain in healthy participants.

Previous physiology-driven pain studies focused on examining the presence or intensity of physical pain. However, people experience various types of pain, including social pain, which induces negative mood; emotional distress; and neural activities associated with physical pain. In particular, comparison of autonomic nervous system (ANS) responses between social and physical pain in healthy adults has not been well demonstrated. We explored the ANS responses induced by two types of pain-social pain, associated with a loss of social ties; and physical pain, caused by a pressure cuff-based on multimodal physiological signals. Seventy-three healthy individuals (46 women; mean age = 20.67 ± 3.27 years) participated. Behavioral responses were assessed to determine their sensitivity to pain stimuli. Electrocardiogram, electrodermal activity, photoplethysmogram, respiration, and finger temperature (FT) were measured, and 12 features were extracted from these signals. Social pain induced increased heart rate (HR) and skin conductance (SC) and decreased blood volume pulse (BVP), pulse transit time (PTT), respiration rate (RR), and FT, suggesting a heterogeneous pattern of sympathetic-parasympathetic coactivation. Moreover, physical pain induced increased heart rate variability (HRV) and SC, decreased BVP and PTT, and resulted in no change in FT, indicating sympathetic-adrenal-medullary activation and peripheral vasoconstriction. These results suggest that changes in HR, HRV indices, RR, and FT can serve as markers for differentiating physiological responses to social and physical pain stimuli.

Online multi-hypergraph fusion learning for cross-subject emotion recognition

Multimodal fusion for emotion recognition has received increasing attention from researchers because of its ability to effectively leverage multimodal complementary information. However, there are two main challenges lead to performance degradation of existing emotion recognition models, which limits the practical use of existing models. One is that multimodal signals are difficult to fuse effectively to respond to the complexity of emotions. The other is that the individual variability and non-stationarity of physiological signals lead to poor performance of the model on new subjects. In particular, existing methods will not work well when faced with emotion recognition of new subjects in online scenarios. In this paper, we propose a novel online multi-hypergraph fusion learning method (OnMHF) to effectively fuse multimodal information and to reduce the difference between training data and test data for online cross-subject emotion recognition. Specifically, in a training phase, a multi-hypergraph fusion is proposed to fuse multimodal physiological signals to effectively obtain emotion-aware information via leveraging multimodal complementary information and high-order correlations among multimodal signals. In an online recognizing phase, an online multi-hypergraph learning is designed to learn online multimodal information from online multimodal data by updating hypergraph structure. As a result, the proposed method can be more effective for emotion recognition of target subjects when target data arrive in an online manner. Experimental results have demonstrated that the proposed method outperforms the baselines and compared state-of-the-art methods in online emotion recognition tasks.

Using XgBoost and multimodal physiological signals to identify flight difficulty

Using XgBoost and multimodal physiological signals to identify flight difficulty

Thermoelectric Hydrogel Electronic Skin for Passive Multimodal Physiological Perception.

Electronic skins (e-skins) are being extensively researched for their ability to recognize physiological data and deliver feedback via electrical signals. However, their wide range of applications is frequently restricted by the indispensableness of external power supplies and single sensory function. Here, we report a passive multimodal e-skin for real-time human health assessment based on a thermoelectric hydrogel. The hydrogel network consists of poly(vinyl alcohol)/low acyl gellan gum with [Fe(CN)6]4-/3- as the redox couple. The introduction of glycerol and Li+ furnishes the gel-based e-skin with antidrying and antifreezing properties, a thermopower of 2.04 mV K-1, fast self-healing in less than 10 min, and high conductivity of 2.56 S m-1. As a prospective application, the e-skin can actively perceive multimodal physiological signals without the need for decoupling, including body temperature, pulse rate, and sweat content, in real time by synergistically coupling sensing and transduction. This work offers a scientific basis and designs an approach to develop passive multimodal e-skins and promotes the application of wearable electronics in advanced intelligent medicine.

Driving Cognitive Alertness Detecting Using Evoked Multimodal Physiological Signals Based on Uncertain Self-Supervised Learning.

Multimodal physiological signals play a pivotal role in drivers' perception of work stress. However, the scarcity of labels and the multitude of modalities render the utilization of physiological signals for driving cognitive alertness detection challenging. We thus propose a multimodal physiological signal detection model based on self-supervised learning. First, in order to mine the intrinsic information of data and enable data to highlight effective information, we introduce a multiscale entropy (MSE) evoked attention mechanism. Secondly, the multimodal patches undergo processing through a novel cascaded attention mechanism. This attention mechanism is rooted in patch-level interactions within each modality, progressively integrating and interacting with other modalities in a cascading manner, thereby mitigating computational complexity. Moreover, a multimodal uncertainty-aware module is devised to effectively cope with intricate variations in the data. This module enhances its generalization ability through the incorporation of uncertain resampling. Experiments were conducted on the DriveDB dataset and the CogPilot dataset with both the linear probing and the fine-tuning evaluation protocols. Experimental results in subject-dependent setting show that our model significantly outperforms previous competitive baselines. In the linear probing evaluation, our model achieves on average 6.26%, 6.64%, and 7.75% improvements in Accuracy (Acc), Recall (Rec), and F1 Score. It also outperforms other models by 7.96% in Acc, 9.13% in Rec, and 9.2% in F1 using the fine-tuning evaluation. Furthermore, our model also demonstrates robust performance in subject-independent setting.

MMPHGCN: A Hypergraph Convolutional Network for Detection of Driver Intention on Multimodal Physiological Signals

MMPHGCN: A Hypergraph Convolutional Network for Detection of Driver Intention on Multimodal Physiological Signals

Advancements in Affective Disorder Detection: Using Multimodal Physiological Signals and Neuromorphic Computing Based on SNNs

Advancements in Affective Disorder Detection: Using Multimodal Physiological Signals and Neuromorphic Computing Based on SNNs

Deep Learning-Based Automated Emotion Recognition Using Multimodal Physiological Signals and Time-Frequency Methods

Deep Learning-Based Automated Emotion Recognition Using Multimodal Physiological Signals and Time-Frequency Methods