Multimodal Biosignals Research Articles

Enhancing Early Detection of Sepsis in Neonates through Multimodal Biosignal Integration: A Study of Pulse Oximetry, Near-Infrared Spectroscopy (NIRS), and Skin Temperature Monitoring.

Sepsis continues to be challenging to diagnose due to its non-specific clinical signs and symptoms, emphasizing the importance of early detection. Our study aimed to enhance the accuracy of sepsis diagnosis by integrating multimodal monitoring technologies with conventional diagnostic methods. The research included a total of 121 newborns, with 39 cases of late-onset sepsis, 35 cases of early-onset sepsis, and 47 control subjects. Continuous monitoring of biosignals, including pulse oximetry (PO), near-infrared spectroscopy (NIRS), and skin temperature (ST), was conducted. An algorithm was then developed in Python to identify early signs of sepsis. The model demonstrated the capability to detect sepsis 6 to 48 h in advance with an accuracy rate of 87.67 ± 7.42%. Sensitivity and specificity were recorded at 76% and 90%, respectively, with NIRS and ST having the most significant impact on predictive accuracy. Despite the promising results, limitations such as sample size, data variability, and potential biases were noted. These findings highlight the critical role of non-invasive biosensing methods in conjunction with conventional biomarkers and cultures, offering a strong foundation for early sepsis detection and improved neonatal care. Further research should be conducted to validate these results across different clinical settings.

Analysis of Multimodal Biosignals during Surprise Conditions Correlates with Psychological Traits

Surprise can simultaneously represent bad or good, pleasant or unpleasant, with the same experiences since understanding how humans' physiological qualities link with their emotional or mental health is required. We conducted quantitative research to concisely correlate mental stress and emotional issues by measuring brain activity, breathing, and heart rate in real time while executing specialized audio-visual stimulation to elicit a surprise event. We evaluated the frequency and temporal domain characteristics to determine if physiological measurements matched biochemical metrics and subjective stress assessments during the elicit surprise condition experiment. We discovered that the brain is still preferable to most in recognizing a human's psychological changes over a short period of time. The temporal (T3) (r = 0.544, p = 0.005) and frontal (Fz) (r = 0.519, p = 0.008) regions were shown to correlate with salivary amylase activity. In comparison to other channels, there was a negative association between stress perception and the occipital site (O1, r = -0.618, p = 0.001). We also found that heart rate variability activity correlates with arousal perception. By looking at specific multimodal biosignals, it is possible to understand human psychological traits by recording specific physiological signals for daily mental health monitoring.

Cerebral infraction prediction system using ECG and PPG bio-signal

Since stroke causes death or serious disability, active primary prevention and early detection of prognostic symptoms are very important. Stroke can be divided into ischemic stroke and hemorrhagic stroke, and they should be minimized by emergency treatment such as thrombolytic or coagulant administration. It is essential to detect in real time the precursor symptoms of stroke, which occur differently for each individual, and to provide professional treatment by a medical institution within the proper treatment window. However, studies have focused on developing acute treatment or clinical treatment guidelines after the onset of stroke rather than detecting the prognostic symptoms of stroke. In particular, studies have mostly used image analysis such as Magnetic Resonance Imaging (MRI) or Computed Tomography (CT) to detect and predict prognostic symptoms in stroke patients. Not only are these methodologies difficult to apply early in real time, but they also have limitations in terms of long testing times and high costs. This paper proposes a system that can predict and semantically interpret stroke prognostic symptoms based on machine learning using multimodal biosignals from Electrocardiogram (ECG) and Photoplethysmogram (PPG). As a result, the real-time prediction of stroke prognosis in elderly patients showed simultaneously high prediction accuracy and performance. Additionally, the CNN-LSTM model using raw data of ECG and PPG demonstrated a satisfactory prediction accuracy of 99.15%.

CNN and Rf based Early Detection of Brain Stroke Using Bio-Electrical Signals

The brain is a vital component of the body that is in control of involuntary and voluntary movements such as walking, memory, and vision. Nowadays, some of the most prevalent brain disorders include Alzheimer's disease, brain tumors, and epilepsy (paralysis or stroke). As a result, stroke has become a significant global health concern, with high rates of mortality and disability. Importantly, approximately two-thirds of all strokes occur in developing countries, highlighting the significant burden of this condition in these regions. Therefore, emphasizing the timely detection and appropriate treatment of brain tumors is crucial. Given the high potential for mortality or severe disability associated with stroke disease, prioritizing active primary prevention and early identification of prognostic symptoms is of paramount importance. Ischemic stroke and hemorrhagic stroke are the two primary classifications for stroke diseases. Each type calls for specific emergency treatments, such as the administration of thrombolytics or coagulants, tailored to their respective underlying mechanisms. However, to effectively manage stroke, it is crucial to promptly identify the precursor symptoms in real-time, as they can vary among individuals. Timely professional treatment within the appropriate treatment window is essential and should be provided by a medical institution. In contrast, prior research has primarily centered around the formulation of acute treatment strategies or clinical guidelines subsequent to the occurrence of a stroke, rather than giving sufficient attention to the early identification of prognostic symptoms. Specifically, recent research has extensively utilized image analysis techniques, such as computed tomography (CT) or magnetic resonance imaging (MRI), as a primary approach for detecting and predicting prognostic symptoms in stroke patients.
 Traditional methodologies not only encounter difficulties in achieving early real-time diagnosis but also exhibit limitations in terms of prolonged testing duration and high testing costs. In this study, we introduce a novel system that employs machine learning techniques to predict and semantically interpret prognostic symptoms of stroke. Our approach utilizes real-time measurement of multi-modal bio-signals, namely electrocardiogram (ECG) and photoplethysmography (PPG), with a specific focus on the elderly population.
 To facilitate real-time prediction of stroke disease during walking, we have developed a stroke disease prediction system that incorporates a hybrid ensemble architecture. This architecture synergistically combines Convolutional Neural Network (CNN) and Random Forest (RF) models, enabling accurate and timely prognostication of stroke disease. The suggested method prioritises the convenience of use of bio-signal sensors for the elderly by collecting bio-signals from three electrodes placed on the index finger. These signals include ECG and PPG, and they are obtained while the participants walk. The CNN-RF model delivers satisfactory prediction accuracy when using raw ECG and PPG data. F1-Score, Sensitivity, Specificity, and Accuracy were the performance parameters used to evaluated the model's performance.

Combining biosignals with RFID to develop a multimodal-shared control interface

The paper presents an approach to develop assistive devices by combining multimodal biosignals and radio frequency identification (RFID). The brain and eye signals have been used as multimodal biosignals to control the movement of a robot in four directions and help reach near the object following a predefined path. RFID shared control over object identification, and the gripper arm connected at the end effector of the robot performs pick and place operations. Horizontal electrooculography (EOG) has been used for x-directional movement control and electroencephalography (EEG) signal obtained by visual stimulus, called steady-state visual-evoked potential (SSVEP) has been used for y-directional movement control of a robot. The SSVEP signal has also been used to ring an alarm in case of an emergency call by the user. Two parameters classification accuracy (CA) and information transfer rate (ITR) have been calculated for the performance evaluation of the proposed multimodal-shared control model and have shown improved results as compared to previous literature. The results also proved that the proposed model can be used for real-time mobility assistive applications.

Reducing Stress with Yoga: A Systematic Review Based on Multimodal Biosignals.

Stress is an enormous concern in our culture because it is the root cause of many health issues. Yoga asanas and mindfulness-based practices are becoming increasingly popular for stress management; nevertheless, the biological effect of these practices on stress reactivity is still a research domain. The purpose of this review is to emphasize various biosignals that reflect stress reduction through various yoga-based practices. A comprehensive synthesis of numerous prior investigations in the existing literature was conducted. These investigations undertook a thorough examination of numerous biosignals. Various features are extracted from these signals, which are further explored to reflect the effectiveness of yoga practice in stress reduction. The multifaceted character of stress and the extensive research undertaken in this field indicate that the proposed approach would rely on multiple modalities. The notable growth of the body of literature pertaining to prospective yoga processes is deserving of attention; nonetheless, there exists a scarcity of research undertaken on these mechanisms. Hence, it is recommended that future studies adopt more stringent yoga methods and ensure the incorporation of suitable participant cohorts.

Assessment of Driver's Stress using Multimodal Biosignals and Regularized Deep Kernel Learning.

In this work, we classify the stress state of car drivers using multimodal physiological signals and regularized deep kernel learning. Using a driving simulator in a controlled environment, we acquire electrocardiography (ECG), electrodermal activity (EDA), photoplethysmography (PPG), and respiration rate (RESP) from N = 10 healthy drivers in experiments of 25min duration with different stress states (5min resting, 10min driving, 10min driving + answering cognitive questions). We manually remove unusable segments and approximately 4h of data remain. Multimodal time and frequency features are extracted and employed to regularized deep kernel machine learning based on a fusion framework. Task-specific representations of different physiological signals are combined using intermediate fusion. Subsequently, the fused multimodal features are fed a support vector machine (SVM) and a random forest (RF) for stress classification. The experimental results show that the proposed approach can discriminate between stress states. The combination of PPG and ECG using RF as classifier yields the highest F1-score of 0.97 in the test set. PPG only and RF yield a maximum F1-score of 0.90. Furthermore, subject-specific cross-validation improves performance. ECG and PPG signals are reliable in classifying the stress state of a car driver. In summary, the proposed framework could be extended to real-time stress state assessment in driving conditions.

Stretchable and Biocompatible Transparent Electrodes for Multimodal Biosignal Sensing from Exposed Skin

AbstractReal‐time health monitoring technology in daily life requires mechanically robust and transparent electrodes for multimodal biosignal sensing from exposed human epidermis. Here, highly stretchable transparent electrodes comprising a water‐dispersed conductive polymer, poly(3,4‐ethylenedioxythiophene):poly(styrenesulfonate) (PEDOT:PSS), and a protic ionic liquid (IL), 3‐methylimidazolium:bis(trifluoromethylsulfonyl)amide (p‐MIM:TFSI) are reported. Owing to the high water miscibility of p‐MIM:TFSI and its favorable ion exchange capability with PEDOT:PSS, PEDOT:PSS/p‐MIM:TFSI transparent electrodes show enhanced electrical conductivity (σ = 450 S cm−1) and thin‐film stretchability represented by crack onset strain (εc) exceeding 50%. These electrodes outperform other PEDOT:PSS electrodes processed with an aprotic counterpart, 1‐ethyl‐3‐methylimidazolium(EMIM):TFSI, or a traditional ionic salt, Li:TFSI. The PEDOT:PSS/p‐MIM:TFSI thin‐film electrodes are also biocompatible and conformally adhere to human skin; therefore, multimodal biosignals including electrocardiogram, electrooculogram, and electromyogram with high signal‐to‐noise ratios from exposed epidermis on human faces and arms under various measurement conditions mimicking daily activities are collected. Considering the importance of light penetration through human skin for stable biological activity during biosignal monitoring, the results can broaden the applicability of daily‐use wearable biosignal sensors by applying them to exposed human skin.

Feature-Based Information Retrieval of Multimodal Biosignals with a Self-Similarity Matrix: Focus on Automatic Segmentation.

Biosignal-based technology has been increasingly available in our daily life, being a critical information source. Wearable biosensors have been widely applied in, among others, biometrics, sports, health care, rehabilitation assistance, and edutainment. Continuous data collection from biodevices provides a valuable volume of information, which needs to be curated and prepared before serving machine learning applications. One of the universal preparation steps is data segmentation and labelling/annotation. This work proposes a practical and manageable way to automatically segment and label single-channel or multimodal biosignal data using a self-similarity matrix (SSM) computed with signals' feature-based representation. Applied to public biosignal datasets and a benchmark for change point detection, the proposed approach delivered lucid visual support in interpreting the biosignals with the SSM while performing accurate automatic segmentation of biosignals with the help of the novelty function and associating the segments grounded on their similarity measures with the similarity profiles. The proposed method performed superior to other algorithms in most cases of a series of automatic biosignal segmentation tasks; of equal appeal is that it provides an intuitive visualization for information retrieval of multimodal biosignals.

Content-based multiple evidence fusion on EEG and eye movements for mild depression recognition

Background and objectiveDepression is a serious neurological disorder that has become a major health problem worldwide. The detection of mild depression is important for the diagnosis of depression in early stages. This research seeks to find a more accurate fusion model which can be used for mild depression detection using Electroencephalography and eye movement data. MethodsThis study proposes a content-based multiple evidence fusion (CBMEF) method, which fuses EEG and eye movement data at decision level. The method mainly includes two modules, the classification performance matrix module and the dual-weight fusion module. The classification performance matrices of different modalities are estimated by Bayesian rule based on confusion matrix and Mahalanobis distance, and the matrices were used to correct the classification results. Then the relative conflict degree of each modality is calculated, and different weights are assigned to the above modalities at the decision fusion layer according to this conflict degree. ResultsThe experimental results show that the proposed method outperforms other fusion methods as well as the single modality results. The highest accuracies achieved 91.12%, and sensitivity, specificity and precision were 89.20%, 93.03%, 92.76%. ConclusionsThe promising results showed the potential of the proposed approach for the detection of mild depression. The idea of introducing the classification performance matrix and the dual-weight model to multimodal biosignals fusion casts a new light on the researches of depression recognition.

AI-Based Stroke Disease Prediction System Using ECG and PPG Bio-Signals

Since stroke disease often causes death or serious disability, active primary prevention and early detection of prognostic symptoms are very important. Stroke diseases can be divided into ischemic stroke and hemorrhagic stroke, and they should be minimized by emergency treatment such as thrombolytic or coagulant administration by type. First, it is essential to detect in real time the precursor symptoms of stroke, which occur differently for each individual, and to provide professional treatment by a medical institution within the proper treatment window. However, prior studies have focused on developing acute treatment or clinical treatment guidelines after the onset of stroke rather than detecting the prognostic symptoms of stroke. In particular, in recent studies, image analysis such as magnetic resonance imaging (MRI) or computed tomography (CT) has mostly been used to detect and predict prognostic symptoms in stroke patients. Not only are these methodologies difficult to diagnose early in real-time, but they also have limitations in terms of a long test time and a high cost of testing. In this paper, we propose a system that can predict and semantically interpret stroke prognostic symptoms based on machine learning using the multi-modal bio-signals of electrocardiogram (ECG) and photoplethysmography (PPG) measured in real-time for the elderly. To predict stroke disease in real-time while walking, we designed and implemented a stroke disease prediction system with an ensemble structure that combines CNN and LSTM. The proposed system considers the convenience of wearing the bio-signal sensors for the elderly, and the bio-signals were collected at a sampling rate of 1,000Hz per second from the three electrodes of the ECG and the index finger for PPG while walking. According to the experimental results, C4.5 decision tree showed a prediction accuracy of 91.56% while RandomForest showed a prediction accuracy of 97.51% during walking by the elderly. In addition, the CNN-LSTM model using raw data of ECG and PPG showed satisfactory prediction accuracy of 99.15%. As a result, the real-time prediction of the elderly stroke patients simultaneously showed high prediction accuracy and performance.

Signal Quality Index Based on Template Cross-Correlation in Multimodal Biosignal Chair for Smart Healthcare.

We investigated the effects of a quality screening method on unconstrained measured signals, including electrocardiogram (ECG), photoplethysmogram (PPG), and ballistocardiogram (BCG) signals, in our collective chair system for smart healthcare. Such an investigation is necessary because unattached or unbound sensors have weaker connections to body parts than do conventional methods. Using the biosignal chair, the physiological signals collected during sessions included a virtual driving task, a physically powered wheelchair drive, and three types of body motions. The signal quality index was defined by the similarity between the observed signals and noise-free signals from the perspective of the cross-correlations of coefficients with appropriate individual templates. The goal of the index was to qualify signals without a reference signal to assess the practical use of the chair in daily life. As expected, motion artifacts have adverse effects on the stability of physiological signals. However, we were able to observe a supplementary relationship between sensors depending on each movement trait. Except for extreme movements, the signal quality and estimated heart rate (HR) remained within the range of criteria usable for status monitoring. By investigating the signal reliability, we were able to confirm the suitability of using the unconstrained biosignal chair to collect real-life measurements to improve safety and healthcare.

Assessing Arousal Through Multimodal Biosignals: A Preliminary Approach.

The increase in Autism Spectrum Disorder (ASD) prevalence estimates over the last decades has driven a quest to develop new forms of rehabilitation that can be accessible to a larger part of this population. These rehabilitation approaches often take the form of computer games that are blind to the user's emotional state, which compromises their efficacy. In this study, a set of physiological signals were acquired in simultaneous with functional Magnetic Resonance Imaging (fMRI) with the future prospect of combining both kinds of data to create models capable of assessing the true emotional state of their users based on physiological response as a measure of autonomic nervous system, having as ground truth the activity of targeted brain regions. This paper describes an initial approach, focusing on the information contained on the physiological signals alone. A total of 35 features were extracted from biosignals' segments and subsequently used for automatic classification of arousal state (High Arousal vs. Low Arousal). The suboptimal results, although some extracted features present statistically significant differences, underline the challenging nature of our proposal and the added obstacles of recording physiological signals in the magnetic resonance environment. Further exploration of the measured signals is needed to gather a bigger number of discriminative features that can improve classification outcomes.

Human–Machine Interaction in Driving Assistant Systems for Semi-Autonomous Driving Vehicles

Currently, the existing vehicle-centric semi-autonomous driving modules do not consider the driver’s situation and emotions. In an autonomous driving environment, when changing to manual driving, human–machine interface and advanced driver assistance systems (ADAS) are essential to assist vehicle driving. This study proposes a human–machine interface that considers the driver’s situation and emotions to enhance the ADAS. A 1D convolutional neural network model based on multimodal bio-signals is used and applied to control semi-autonomous vehicles. The possibility of semi-autonomous driving is confirmed by classifying four driving scenarios and controlling the speed of the vehicle. In the experiment, by using a driving simulator and hardware-in-the-loop simulation equipment, we confirm that the response speed of the driving assistance system is 351.75 ms and the system recognizes four scenarios and eight emotions through bio-signal data.

Real-Time Proxy-Control of Re-Parameterized Peripheral Signals using a Close-Loop Interface.

The fields that develop methods for sensory substitution and sensory augmentation have aimed to control external goals using signals from the central nervous systems (CNS). Less frequent however, are protocols that update external signals self-generated by interactive bodies in motion. There is a paucity of methods that combine the body-heart-brain biorhythms of one moving agent to steer those of another moving agent during dyadic exchange. Part of the challenge to accomplish such a feat has been the complexity of the setup using multimodal bio-signals with different physical units, disparate time scales and variable sampling frequencies. In recent years, the advent of wearable bio-sensors that can non-invasively harness multiple signals in tandem, has opened the possibility to re-parameterize and update the peripheral signals of interacting dyads, in addition to improving brain- and/or body-machine interfaces. Here we present a co-adaptive interface that updates efferent somatic-motor output (including kinematics and heart rate) using biosensors; parameterizes the stochastic bio-signals, sonifies this output, and feeds it back in re-parameterized form as visuo/audio-kinesthetic reafferent input. We illustrate the methods using two types of interactions, one involving two humans and another involving a human and its avatar interacting in near real time. We discuss the new methods in the context of possible new ways to measure the influences of external input on internal somatic-sensory-motor control.

Deep Autoencoder for Identification of Abnormal Gait Patterns Based on Multimodal Biosignals

Deep Autoencoder for Identification of Abnormal Gait Patterns Based on Multimodal Biosignals

Drivers’ Affect Recognition System Utilizing Multimodal Biosignals

Drivers’ Affect Recognition System Utilizing Multimodal Biosignals

Using Multimodal Bio-Signals for Prediction of Physiological Cognitive State Under Free-Living Conditions

Drowsiness is a common human physiological response. Research suggests that insufficient sleep results in low energy levels, various negative physiological and psychological effects on the body, and abnormal cognitive functioning. Studies have predominantly focused on driving while drowsy, using brainwave measurement and facial detection techniques to address this topic, whereas few have discussed the physiological prediction of drowsiness. In addition to driving, working conditions and environments as well as daily activities also correspond to the risk of accidents occurring when people are drowsy. This study designed an experiment consisting of five tests in which a brainwave sensor, eye tracker, heart rate sensor, and galvanic skin response sensor were used to record physiological changes in participants. The data indicated the binary outcomes of patients either being or not being in a state of drowsiness or sleepiness. During various states of drowsiness or sleepiness, brain wave activity, eye movement, heart rate, and GSR were measured, and differences in these physiological responses were analyzed. A classification model was also used to predict a participant’s state of drowsiness or sleepiness. Data on physiological characteristics included eye movement and the heart rate value, which was calculated during various states of drowsiness or sleepiness to obtain a value. Brain wave and GSR signals were converted through software development kit (SDK) programming at the sensor end. Subsequently, the data were processed through an artificial neural network (ANN), back propagation network, and support vector machine (SVM). In the experiment, the SVM hyperparameters were adjusted, the ANN model was added to the Adam optimization model, and overfitting was avoided to ensure that the results were comprehensive. According to the experiment results, the use of SVM yields the optimal classification performance, reaching an accuracy of 89.1%; 90% of participants were also categorized more accurately through SVM than ANN.

A Novel Fuzzy Rough Nearest Neighbors Emotion Recognition Approach Based on Multimodal Wearable Biosensor Network

Due to the diversity and complexity of emotional bioinformation, most emotion recognition studies relied to ponderous medical grade electroencephalography (EEG) measuring devices. The emotion recognition scenarios were limited to hospitals or laboratories. It is hard to directly apply these research achievements into the emotion monitoring in daily-life. In this paper, a novel emotion recognition approach based on multimodal wearable biosensor network is investigated. In order to facilitate emotion monitoring in daily-life, a multimodal wearable biosensor network is constructed. The multimodal bio-signals are acquired by wearable biosensor measuring nodes, and then transmitted to sink node through wireless communication technologies. According to the fuzzy and rough characteristics of human emotions, the fuzzy rough nearest neighbors (FRNN) algorithm is introduced to classify different emotions. By considering the fuzzy thresholds of EEG concentration, a novel FRNN emotion recognition approach is proposed. The proposed method narrows the classification range of samples with significant concentration and reduces the disturbance of noisy samples, so that the high accuracy (65.6%) and fast speed were achieved in wearable scenario. Experiments verified the effectiveness of the proposed approach.

Deep Learning based Emotion Classification using Multi Modal Bio-signals

Deep Learning based Emotion Classification using Multi Modal Bio-signals