Multimodal Sensor Research Articles

AI System for Autonomous Vehicles

The rapid advancement of artificial intelligence (AI) has revolutionized the development of autonomous vehicles, offering transformative potential for the future of transportation. This research investigates the implementation of AI-driven algorithms in autonomous vehicles, focusing on their ability to enhance decision-making, navigation, and safety. By employing state-of-the-art machine learning models, including deep learning and reinforcement learning, the study explores how these technologies can optimize real-time processing of sensor data, environmental perception, and adaptive control mechanisms. The findings demonstrate that AI algorithms can significantly improve the accuracy of object detection, trajectory prediction, and path planning, thereby reducing the likelihood of collisions and enhancing overall road safety. A key contribution of this work is the integration of a multi-modal sensor fusion approach, combining data from LiDAR, cameras, radar, and GPS to create a comprehensive and reliable understanding of the vehicle’s surroundings. Additionally, the research highlights the role of AI in enabling autonomous vehicles to learn from vast amounts of driving data, facilitating continuous improvement and adaptability in diverse driving conditions. The implications of this study are profound, suggesting that AI-powered autonomous vehicles could lead to safer, more efficient, and environmentally sustainable transportation systems. However, the research also identifies challenges related to computational complexity, real-time decision-making, and ethical considerations in AI-driven autonomy. Future work will focus on addressing these challenges and exploring the broader societal impacts of widespread autonomous vehicle adoption.

Radiation synthesis of high conductivity hydrogel based on tragacanth gum/poly (ionic liquids) for multimodal sensors and supercapacitor

Natural polymer-based hydrogels have found extensive use in flexible sensing, energy storage, and other fields because of their environmental sustainability and biocompatibility. Nonetheless, numerous challenges persist in the development of hydrogels with outstanding conductivity solely from natural polymers. Herein, we have successfully synthesized hydrogels based on natural polymer (tragacanth gum) and ionic liquids (1-vinyl-3-ethylimidazolium bromide) using a convenient and efficient one-step ionizing radiation method (TG/PIL hydrogels). The TG/PIL hydrogels exhibit high ionic conductivity (7.1 S m−1 at 25 °C), and can be used for multimodal sensors, including strain and temperature sensors. It has exceptional capabilities in monitoring human motor behavior, capturing subtle facial expressions and pulses beat. TG/PIL hydrogel can also accurately sense changes in the temperature of the external environment, and have significant thermal sensitivity within the range of 40 to 60 °C (−3.22 % /°C). Furthermore, the high conductivity of TG/PIL hydrogels enables them to exhibit outstanding performance in supercapacitor electrolytes, it has good stability in a certain load bearing range, temperature range, folding angle range. This work offers a straightforward technique for creating a multimodal hydrogel sensor, with promising applications in flexible wearable devices, energy storage, and beyond.

Rolling bearing fault diagnosis based on acoustic-vibration data fusion and mode decomposition combined with the crested porcupine optimization algorithm

Variational Mode Decomposition (VMD) is extensively utilized in the domain of rotating machinery fault diagnosis. Nevertheless, the reliance on empirical parameter tuning and the limitations inherent in traditional single-signal (either vibration signals or acoustic signals) fault diagnosis methods present substantial challenges. These include incomplete information and high sensitivity to noise. To address these shortcomings, a new method is introduced that integrates multi-modal sensor data and implements an adaptive VMD approach, enhanced through the Crested Porcupine Optimization Algorithm (CPO) to automatically optimize the key parameter. Then, uses optimized VMD to decompose the time series into intrinsic mode functions of different frequencies to capture the multi-scale characteristics of data. Finally, the decomposed acoustic-vibro signal components are fed into the CNN-BiGRU-AT network, which learns the spatio-temporal dependencies to enhance the accuracy of fault classification. The experimental results conducted across diverse high-noise environments demonstrate that this method’s superior noise robustness compared to single-modality sensor approaches， highlighting the method's potential for machinery fault detection in noisy conditions.

Oriented Alginate-Poly(vinyl alcohol) Electrospun Nanofibers for Multimodal Sensing and Gesture Language Recognition.

Flexible nanofiber sensors have gained substantial attention in extending application scenarios owing to their desirable lightweight, comfort, and breathability. Nevertheless, disorder and uneven dimension issues of nanofibers are the leading concerns in their multifunctional response, which often leads to erratic response signals as well as poor linearity. In this work, a high-performance oriented nanofiber film with a three-dimensional network consisting of alginate sodium, poly(vinyl alcohol), and poly(ethylene oxide) was successfully fabricated by a controllable directional electrospinning technique. The main properties of the nanofibers are capable of being regulated intentionally by varying the electrospinning temperature, collector rotation speed, and polymer concentrations. Based on the favorable structure orientation, the nanofiber film displays satisfied biodegradability and high mechanical strength (575.1 MPa). Being integrated with modified magnetic particles, the sensors not only display a fast response speed, high magnetic sensitivity, and exceptional recoverability in response to magnetic fields but also show favorable sensitivities and reliable long-term durability under mechanical excitations. As a wearable sensor, it can accurately perceive the physiological signals generated by the human body in real-time. Furthermore, with the assistance of a convolutional neural network model, a gesture language recognition system is developed by integrating multiple sensors to realize a high recognition accuracy (∼99.08%). This study provides a feasible strategy to manufacture high-performance multimodal sensors for wearable human-machine interaction applications.

Sweat‐powered, skin‐adhesive multimodal sensor for long‐term and real‐time sweat monitoring

AbstractThe importance of continuous healthcare management has significantly accelerated the development of wearable devices for monitoring health‐related physical and biochemical markers. Despite extensive research on wearable devices for physiological and biochemical monitoring, critical issues of power management and device/skin interfacial properties restrict the advancement of personalized healthcare and early disease detection. Here, we report a multimodal sweat monitoring device featuring a real‐time display and long‐term data analysis based on self‐powered format of sweat‐activated batteries (SABs). The polyvinyl alcohol‐sucrose (PVA‐Suc) hydrogel serves as the key component for the SAB, offering not only great long‐term adhesive properties for conformable wearability but also significant power generation capabilities. A maximum current density of 44.06 mA cm−2 and a maximum power density of 21.89 mW cm−2 can be realized for the hydrogel based SAB. The resulting device integrates an advanced colorimetric and electrochemical sensor array to measure pH levels, glucose concentrations, and chloride ion levels in human sweat, with data wirelessly transmitted by Near Field Communication. The self‐powering features and multiple mode sensing function offer sufficient power to support real‐time monitoring of metabolic biomarkers in sweat, with the ability to visually observe changes in the colorimetric sensors for long‐term data monitoring.

In-Bed Monitoring: A Systematic Review of the Evaluation of In-Bed Movements Through Bed Sensors

The growing popularity of smart beds and devices for remote healthcare monitoring is based on advances in artificial intelligence (AI) applications. This systematic review aims to evaluate and synthesize the growing literature on the use of machine learning (ML) techniques to characterize patient in-bed movements and bedsore development. This review is conducted according to the principles of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and is registered in the International Prospective Register of Systematic Reviews (PROSPERO CRD42022314329). The search was performed through nine scientific databases. The review included 78 articles, including 142 ML models. The applied ML models revealed significant heterogeneity in the various methodologies used to identify and classify patient behaviors and postures. The assortment of ML models encompassed artificial neural networks, deep learning architectures, and multimodal sensor integration approaches. This review shows that the models for analyzing and interpreting in-bed movements perform well in experimental settings. Large-scale real-life studies are lacking in diverse patient populations.

A Highly Sensitive, Conductive, and Flexible Hydrogel Sponge as a Discriminable Multimodal Sensor for Deep‐Learning‐Assisted Gesture Language Recognition

AbstractFlexible multimodal sensors have gained increasing popularity for applications in healthcare and extreme environment operations owing to their all‐around environmental perception and data acquisition capabilities. However, fabricating a magnetism‐mechanics‐humidity multimodal sensor that possesses high sensitivity without signal overlapping while in a facile methodology remains a great challenge. Herein, a highly sensitive, conductive, and flexible hydrogel sponge sensor with discriminable magnetism, mechanics, and humidity sensing capability is proposed, which shows stable pore size (19.30 µm) and satisfactory mechanical properties based on the synergistic hydrogen bonding among sodium alginate, poly(vinyl alcohol) and glycerol. The proposed sensors can not only display favorable humidity sensing ability with rapid response/recovery time (2.5/4 s) but also possess enhanced sensitivities (a gauge factor of 0.46 T−1 for magnetic field, −1.16 kPa−1 for pressure), superior stability and durability (over 8000 cycles). Benefiting from the separated capacitive and resistive response signals, the sensors can precisely distinguish the magnetic, mechanical, and humidity stimuli without cross‐talk. Further, the sensor arrays assisted by the deep learning algorithm are developed to realize gesture language recognition with a high accuracy of 99.17%. It can be believed that this high‐performance sensor will have good prospects in future soft electronics and human‐machine interaction systems.

Feasibility of snapshot testing using wearable sensors to detect cardiorespiratory illness (COVID infection in India).

The COVID-19 pandemic has challenged the current paradigm of clinical and community-based disease detection. We present a multimodal wearable sensor system paired with a two-minute, movement-based activity sequence that successfully captures a snapshot of physiological data (including cardiac, respiratory, temperature, and percent oxygen saturation). We conducted a large, multi-site trial of this technology across India from June 2021 to April 2022 amidst the COVID-19 pandemic (Clinical trial registry name: International Validation of Wearable Sensor to Monitor COVID-19 Like Signs and Symptoms; NCT05334680; initial release: 04/15/2022). An Extreme Gradient Boosting algorithm was trained to discriminate between COVID-19 infected individuals (n = 295) and COVID-19 negative healthy controls (n = 172) and achieved an F1-Score of 0.80 (95% CI = [0.79, 0.81]). SHAP values were mapped to visualize feature importance and directionality, yielding engineered features from core temperature, cough, and lung sounds as highly important. The results demonstrated potential for data-driven wearable sensor technology for remote preliminary screening, highlighting a fundamental pivot from continuous to snapshot monitoring of cardiorespiratory illnesses.

Retraction Note: Application of optical motion capture based on multimodal sensors in badminton player motion recognition system

Retraction Note: Application of optical motion capture based on multimodal sensors in badminton player motion recognition system

A Wireless and Wearable Multimodal Sensor to Non-Invasively Monitor Transabdominal Placental Oxygen Saturation and Maternal Physiological Signals.

Poor placental development and placental defects can lead to adverse pregnancy outcomes such as pre-eclampsia, fetal growth restriction, and stillbirth. This study introduces two sensors, which use a near-infrared spectroscopy (NIRS) technique to measure placental oxygen saturation transabdominally. The first one, an NIRS sensor, is a wearable device consisting of multiple NIRS channels. The second one, a Multimodal sensor, which is an upgraded version of the NIRS sensor, is a wireless and wearable device, integrating a motion sensor and multiple NIRS channels. A pilot clinical study was conducted to assess the feasibility of the two sensors in measuring transabdominal placental oxygenation in 36 pregnant women (n = 12 for the NIRS sensor and n = 24 for the Multimodal sensor). Among these subjects, 4 participants had an uncomplicated pregnancy, and 32 patients had either maternal pre-existing conditions/complications, neonatal complications, and/or placental pathologic abnormalities. The study results indicate that the patients with maternal complicated conditions (69.5 ± 5.4%), placental pathologic abnormalities (69.4 ± 4.9%), and neonatal complications (68.0 ± 5.1%) had statistically significantly lower transabdominal placental oxygenation levels than those with an uncomplicated pregnancy (76.0 ± 4.4%) (F (3,104) = 6.6, p = 0.0004). Additionally, this study shows the capability of the Multimodal sensor in detecting the maternal heart rate and respiratory rate, fetal movements, and uterine contractions. These findings demonstrate the feasibility of the two sensors in the real-time continuous monitoring of transabdominal placental oxygenation to detect at-risk pregnancies and guide timely clinical interventions, thereby improving pregnancy outcomes.

Shuffled ECA-Net for stress detection from multimodal wearable sensor data

BackgroundRecently, stress has been recognized as a key factor in the emergence of individual and social issues. Numerous attempts have been made to develop sensor-augmented psychological stress detection techniques, although existing methods are often impractical or overly subjective. To overcome these limitations, we acquired a dataset utilizing both wireless wearable multimodal sensors and salivary cortisol tests for supervised learning. We also developed a novel deep neural network (DNN) model that maximizes the benefits of sensor fusion. MethodWe devised a DNN involving a shuffled efficient channel attention (ECA) module called a shuffled ECA-Net, which achieves advanced feature-level sensor fusion by considering inter-modality relationships. Through an experiment involving salivary cortisol tests on 26 participants, we acquired multiple bio-signals including electrocardiograms, respiratory waveforms, and electrogastrograms in both relaxed and stressed mental states. A training dataset was generated from the obtained data. Using the dataset, our proposed model was optimized and evaluated ten times through five-fold cross-validation, while varying a random seed. ResultsOur proposed model achieved acceptable performance in stress detection, showing 0.916 accuracy, 0.917 sensitivity, 0.916 specificity, 0.914 F1-score, and 0.964 area under the receiver operating characteristic curve (AUROC). Furthermore, we demonstrated that combining multiple bio-signals with a shuffled ECA module can more accurately detect psychological stress. ConclusionsWe believe that our proposed model, coupled with the evidence for the viability of multimodal sensor fusion and a shuffled ECA-Net, would significantly contribute to the resolution of stress-related issues.

Fine-Grained Recognition of Manipulation Activities on Objects via Multi-Modal Sensing

Fine-Grained Recognition of Manipulation Activities on Objects via Multi-Modal Sensing

Real-time personal healthcare data analysis using edge computing for multimodal wearable sensors

Real-time personal healthcare data analysis using edge computing for multimodal wearable sensors

Wearable network for multilevel physical fatigue prediction in manufacturing workers.

Manufacturing workers face prolonged strenuous physical activities, impacting both financial aspects and their health due to work-related fatigue. Continuously monitoring physical fatigue and providing meaningful feedback is crucial to mitigating human and monetary losses in manufacturing workplaces. This study introduces a novel application of multimodal wearable sensors and machine learning techniques to quantify physical fatigue and tackle the challenges of real-time monitoring on the factory floor. Unlike past studies that view fatigue as a dichotomous variable, our central formulation revolves around the ability to predict multilevel fatigue, providing a more nuanced understanding of the subject's physical state. Our multimodal sensing framework is designed for continuous monitoring of vital signs, including heart rate, heart rate variability, skin temperature, and more, as well as locomotive signs by employing inertial motion units strategically placed at six locations on the upper body. This comprehensive sensor placement allows us to capture detailed data from both the torso and arms, surpassing the capabilities of single-point data collection methods. We developed an innovative asymmetric loss function for our machine learning model, which enhances prediction accuracy for numerical fatigue levels and supports real-time inference. We collected data on 43 subjects following an authentic manufacturing protocol and logged their self-reported fatigue. Based on the analysis, we provide insights into our multilevel fatigue monitoring system and discuss results from an in-the-wild evaluation of actual operators on the factory floor. This study demonstrates our system's practical applicability and contributes a valuable open-access database for future research.

STIM1 functions as a proton sensor to coordinate cytosolic pH with store-operated calcium entry

The meticulous regulation of intracellular pH (pHi) is crucial for maintaining cellular function and homeostasis, impacting physiological processes such as heart rhythm, cell migration, proliferation, and differentiation. Dysregulation of pHi is implicated in various pathologies such as arrhythmias, cancer, and neurodegenerative diseases. Here, we explore the role of STIM1, an ER calcium (Ca2+) sensor mediating Store Operated Ca2+ Entry (SOCE), in sensing pHi changes. Our study reveals that STIM1 functions as a sensor for pHi changes, independent of its Ca2+-binding state. Through comprehensive experimental approaches including confocal microscopy, FRET-based sensors, and mutagenesis, we demonstrate that changes in pHi induce conformational alterations in STIM1, thereby modifying its subcellular localization and activity. We identify two conserved histidine within STIM1 essential for sensing pHi shifts. Moreover, intracellular alkalization induced by agents such as Angiotensin II or NH4Cl enhances STIM1-mediated SOCE, promoting cardiac hypertrophy. These findings reveal a novel facet of STIM1 as a multi-modal stress sensor that coordinates cellular responses to both Ca2+ and pH fluctuations. This dual functionality underscores its potential as a therapeutic target for diseases associated with pH and Ca2+ dysregulation.

Data Quality Based Intelligent Instrument Selection with Security Integration

We propose a novel Data Quality with Security (DQS) integrated instrumentation selection approach that facilitates aggregation of multi-modal data from heterogeneous sources. As our major contribution, we develop a framework that incorporates multiple levels of integration in finding the best DQS-based instrument selection: data fusion from multi-modal sensors embedded into heterogeneous platforms, using multiple quality and security metrics and knowledge integration. Our design addresses the security aspect in the instrumentation design, which is commonly overlooked in real applications, by aggregating it with other metrics into an integral DQS calculus. We develop DQS calculus that formalizes the problem of finding the optimal DQS value. We then propose a Genetic Algorithm–based solution to find an optimal set of sensors in terms of the DQS they provide, while maintaining the level of platform security desirable by the user. We show that our proposed algorithm demonstrates optimal real-time performance in multi-platform instrument selection. To facilitate the framework application by the instrumentation designers and users, we develop and make available multiple Android applications.

A smart sponge with pressure–temperature dual-mode sensing for packaging and transportation

Conventional sponges are limited to providing mechanical protection and lack the capability to monitor the real-time status of transported object. Consequently, there is an urgent need to develop smart sponges that can simultaneously protect and monitor the status of objects during transportation. In this study, we have developed a poly(3,4-ethylenedioxythiophene):polystyrene sulfonate (PEDOT:PSS) melamine sponge (PEDOT:PSS@MS) using a dip coating-centrifugation method. The PEDOT:PSS@MS functions as a dual-mode sensor for pressure and temperature. The sensor has a maximum linear pressure sensitivity of 0.411 kPa−1 within 42.7 kPa, and a fast recovery time of 46 ms. On the other hand, the sensor exhibits a linear temperature sensitivity of 9.42 μV/K via the Seebeck effect in the range of 30 to 80 °C. By integrating multiple PEDOT:PSS@MS units into a 3D-printed packing box, the pressure and temperature orientation distribution of the transported objects can be monitored in real-time. The development of these smart sponges enhances multimodal sensors for linear response performance, holding great potential in intelligent logistics security and advanced packaging solutions.

Crosstalk-free hybrid integrated multimodal sensor for human temperature, humidity, and pressure monitoring

Crosstalk-free hybrid integrated multimodal sensor for human temperature, humidity, and pressure monitoring

Comparative Assessment of Multimodal Sensor Data Quality Collected Using Android and iOS Smartphones in Real-World Settings

Healthcare researchers are increasingly utilizing smartphone sensor data as a scalable and cost-effective approach to studying individualized health-related behaviors in real-world settings. However, to develop reliable and robust digital behavioral signatures that may help in the early prediction of the individualized disease trajectory and future prognosis, there is a critical need to quantify the potential variability that may be present in the underlying sensor data due to variations in the smartphone hardware and software used by large population. Using sensor data collected in real-world settings from 3000 participants’ smartphones for up to 84 days, we compared differences in the completeness, correctness, and consistency of the three most common smartphone sensors—the accelerometer, gyroscope, and GPS— within and across Android and iOS devices. Our findings show considerable variation in sensor data quality within and across Android and iOS devices. Sensor data from iOS devices showed significantly lower levels of anomalous point density (APD) compared to Android across all sensors (p < 1 × 10−4). iOS devices showed a considerably lower missing data ratio (MDR) for the accelerometer compared to the GPS data (p < 1 × 10−4). Notably, the quality features derived from raw sensor data across devices alone could predict the device type (Android vs. iOS) with an up to 0.98 accuracy 95% CI [0.977, 0.982]. Such significant differences in sensor data quantity and quality gathered from iOS and Android platforms could lead to considerable variation in health-related inference derived from heterogenous consumer-owned smartphones. Our research highlights the importance of assessing, measuring, and adjusting for such critical differences in smartphone sensor-based assessments. Understanding the factors contributing to the variation in sensor data based on daily device usage will help develop reliable, standardized, inclusive, and practically applicable digital behavioral patterns that may be linked to health outcomes in real-world settings.

Development of a Machine Learning Framework for Real-Time PMI (Post-Mortem Interval) Estimation in Field Forensics

Accurate estimation of the Post-Mortem Interval (PMI) is critical in forensic investigations, aiding in determining the time of death. However, traditional PMI estimation methods, often reliant on physiological observations and environmental factors, face significant limitations in accuracy and efficiency, especially in field conditions. This paper presents the development of a machine learning (ML) framework designed for real-time PMI estimation, integrating multimodal sensor data to address the challenges encountered in field forensics. Our framework utilizes environmental and physiological features, including body temperature, ambient humidity, and biochemical decomposition markers, to predict PMI with high precision. The ML model, trained on historical forensic data, is deployed on a real-time processing platform, enabling rapid analysis and decision-making in resource- constrained environments. The system is optimized for field operations, incorporating low-power hardware and edge computing capabilities to provide forensic investigators with reliable PMI estimates on-site. Through a series of controlled experiments simulating forensic scenarios, our framework demonstrates a significant improvement in PMI accuracy compared to traditional methods, while maintaining low latency for real-time applications. This research highlights the potential of machine learning to revolutionize forensic practices, offering a scalable and adaptive solution for time-sensitive investigations. Here are some relevant keywords for the development of a machine learning framework for real-time PMI (Post- Mortem Interval) estimation in field forensics: Keywords: Field Forensics, Real-Time Machine Learning, Body Decomposition Stages, Machine Learning in Forensic Science, Artificial Intelligence for PMI Analysis, Sensor Data in PMI Estimation, Deep Learning for PMI Estimation, Automated Forensic Analysis, Data Acquisition in Field Forensics.