Multimodal Sensor Research Articles

Hybrid multimodal wearable sensors for comprehensive health monitoring

Hybrid multimodal wearable sensors for comprehensive health monitoring

Development of Multimodal Sensors for Calcium-Based Food Adulterant Using Tannic Acid Derived Functional Nanomaterials

Development of Multimodal Sensors for Calcium-Based Food Adulterant Using Tannic Acid Derived Functional Nanomaterials

Nitrogen-vacancy centers as a self-gauged micro-scale heater and its application for multi-modal sensing

The optical excitation of nitrogen-vacancy (NV) color centers in diamonds mostly results in fluorescence emission. During this process, a portion of the incident energy is transferred to phonon vibration, which heats the diamond crystal. For single NV color centers, the heat generated by the optical cycle is negligible, while for an ensemble of NV defects, the generated heat accumulates rapidly and heats the diamond. The temperature rise is rapid due to the high thermal conductivity of the diamond. In addition to the ability to be heated by light, the NV defect's unique properties also allow for the precise measurement of temperature using optically detected magnetic resonance. Here, we experimentally demonstrate that microcrystalline diamond containing NV center ensembles can be used as a self-gauged microheater. We attached a microcrystal diamond to an optical fiber in an endoscope configuration, evaluated its performance as a self-gauged heater under varied biologically relevant environments, and discussed its potential applications. In addition to the aforementioned capabilities, the NV defect enables the precise measurement of local magnetic fields. This provides a unique multimodal sensor to probe temperature-controlled magnetic phenomena at microscopic scales.

Optimal Sensor Placement and Multimodal Fusion for Human Activity Recognition in Agricultural Tasks

This study examines the impact of sensor placement and multimodal sensor fusion on the performance of a Long Short-Term Memory (LSTM)-based model for human activity classification taking place in an agricultural harvesting scenario involving human-robot collaboration. Data were collected from twenty participants performing six distinct activities using five wearable inertial measurement units placed at various anatomical locations. The signals collected from the sensors were first processed to eliminate noise and then input into an LSTM neural network for recognizing features in sequential time-dependent data. Results indicated that the chest-mounted sensor provided the highest F1-score of 0.939, representing superior performance over other placements and combinations of them. Moreover, the magnetometer surpassed the accelerometer and gyroscope, highlighting its superior ability to capture crucial orientation and motion data related to the investigated activities. However, multimodal fusion of accelerometer, gyroscope, and magnetometer data showed the benefit of integrating data from different sensor types to improve classification accuracy. The study emphasizes the effectiveness of strategic sensor placement and fusion in optimizing human activity recognition, thus minimizing data requirements and computational expenses, and resulting in a cost-optimal system configuration. Overall, this research contributes to the development of more intelligent, safe, cost-effective adaptive synergistic systems that can be integrated into a variety of applications.

A LiDAR-Camera Joint Calibration Algorithm Based on Deep Learning.

Multisensor (MS) data fusion is important for improving the stability of vehicle environmental perception systems. MS joint calibration is a prerequisite for the fusion of multimodality sensors. Traditional calibration methods based on calibration boards require the manual extraction of many features and manual registration, resulting in a cumbersome calibration process and significant errors. A joint calibration algorithm for a Light Laser Detection and Ranging (LiDAR) and camera is proposed based on deep learning without the need for other special calibration objects. A network model constructed based on deep learning can automatically capture object features in the environment and complete the calibration by matching and calculating object features. A mathematical model was constructed for joint LiDAR-camera calibration, and the process of sensor joint calibration was analyzed in detail. By constructing a deep-learning-based network model to determine the parameters of the rotation matrix and translation matrix, the relative spatial positions of the two sensors were determined to complete the joint calibration. The network model consists of three parts: a feature extraction module, a feature-matching module, and a feature aggregation module. The feature extraction module extracts the image features of color and depth images, the feature-matching module calculates the correlation between the two, and the feature aggregation module determines the calibration matrix parameters. The proposed algorithm was validated and tested on the KITTI-odometry dataset and compared with other advanced algorithms. The experimental results show that the average translation error of the calibration algorithm is 0.26 cm, and the average rotation error is 0.02°. The calibration error is lower than those of other advanced algorithms.

Multi-Modal Sensing Ionogels with Tunable Mechanical Properties and Environmental Stability for Aquatic and Atmospheric Environments.

Ionogels have garnered significant interest due to their great potential in flexible iontronic devices. However, their limited mechanical tunability and environmental intolerance have posed significant challenges for their integration into next-generation flexible electronics in different scenarios. Herein, the synergistic effect of cation-oxygen coordination interaction and hydrogen bonding is leveraged to construct a 3D supramolecular network, resulting in ionogels with tunable modulus, stretchability, and strength, achieving an unprecedented elongation at break of 10800%. Moreover, the supramolecular network endows the ionogels with extremely high fracture energy, crack insensitivity, and high elasticity. Meanwhile, the high environmental stability and hydrophobic network of the ionogels further shield them from the unfavorable effects of temperature variations and water molecules, enabling them to operate within a broad temperature range and exhibit robust underwater adhesion. Then, the ionogel is assembled into a wearable sensor, demonstrating its great potential in flexible sensing (temperature, pressure, and strain) and underwater signal transmission. This work can inspire the applications of ionogels in multifunctional sensing and wearable fields.

A 2-D Vigilance Estimation Method for High-Speed Rail Drivers With Multimodal Sensors

A 2-D Vigilance Estimation Method for High-Speed Rail Drivers With Multimodal Sensors

Moisture-wicking Janus-structure electronic knitted fabric for multimodal wearable mechanical sensing

Multimodal mechanical sensors that can recognize bending direction and exhibit excellent wearable performance are crucial for the fields of physical exercise and healthcare. This study fabricated a knitted multimodal sensor with porous and wetting gradients using hydrophobic core-spun yarns and hydrophilic CNT/WPU conductive yarns in plain stitch and partial tuck stitch. The electronic fabric demonstrated reliable strain sensing unaffected by compression disturbances and displayed a selective response to directional bending. The sensor exhibits a 50% sensing range with a GFmax of −4.41 during strain deformation detection. In terms of bending detection, one side achieves a 10% sensing range (−51.68°) with a GFmax of −6.89, while the other side achieves a 20% sensing range (73.72°) with a GFmax of 8.64. Encouragingly, deep learning incorporated with the sensor achieves an impressive recognition rate of 98.16% for mixed signals with different force levels. Furthermore, it possesses excellent wearable properties such as breathability, moisture permeability, perspiration remove, quick-drying and washability, suitable for applications like activity monitoring, medical rehabilitation evaluation, and gesture recognition. This work presents a promising pathway for the advancement of next-generation multifunctional electronic textiles.

Soft Multimodal Sensors with Decoupled Multimodality and Minimal Wiring for Wearable Systems

AbstractSoft multimodal sensors represent a promising approach for applications in wearable systems and skin‐adhesive devices, benefiting from their flexibility and body compatibility. However, the complexity of sensor wiring design poses challenges while employing multimodal decoupling. Although recent studies have applied various methods, such as adjusting the electrical properties of materials, these methods encounter limitations in signal decoupling when multichannel measurements are implemented. In this study, a signal measurement method utilizing frequency response analysis is developed using a multi‐band band‐stop filter (MBBSF) for multiple multimodal sensors. This approach enables the decoupling of different types of sensors and allows for the simultaneous collection of sensor data through a two‐wire connection. Consequently, it facilitates the concurrent and independent real‐time assessment of strain and force in each sensor. A sensor glove is developed incorporating multiple multimodal sensors with a two‐wire connection to measure grip posture and force. This glove is used effectively to classify unknown objects, thereby demonstrating its practical utility. This practical application provides a promising solution to the complex wiring challenges commonly encountered in multi‐modal sensor wearable systems.

AEPF: Attention-Enabled Point Fusion for 3D Object Detection.

Current state-of-the-art (SOTA) LiDAR-only detectors perform well for 3D object detection tasks, but point cloud data are typically sparse and lacks semantic information. Detailed semantic information obtained from camera images can be added with existing LiDAR-based detectors to create a robust 3D detection pipeline. With two different data types, a major challenge in developing multi-modal sensor fusion networks is to achieve effective data fusion while managing computational resources. With separate 2D and 3D feature extraction backbones, feature fusion can become more challenging as these modes generate different gradients, leading to gradient conflicts and suboptimal convergence during network optimization. To this end, we propose a 3D object detection method, Attention-Enabled Point Fusion (AEPF). AEPF uses images and voxelized point cloud data as inputs and estimates the 3D bounding boxes of object locations as outputs. An attention mechanism is introduced to an existing feature fusion strategy to improve 3D detection accuracy and two variants are proposed. These two variants, AEPF-Small and AEPF-Large, address different needs. AEPF-Small, with a lightweight attention module and fewer parameters, offers fast inference. AEPF-Large, with a more complex attention module and increased parameters, provides higher accuracy than baseline models. Experimental results on the KITTI validation set show that AEPF-Small maintains SOTA 3D detection accuracy while inferencing at higher speeds. AEPF-Large achieves mean average precision scores of 91.13, 79.06, and 76.15 for the car class's easy, medium, and hard targets, respectively, in the KITTI validation set. Results from ablation experiments are also presented to support the choice of model architecture.

Nanocomposite Multimodal Sensor Array Integrated with Auxetic Structure for an Intelligent Biometrics System.

A multimodal sensor array, combining pressure and proximity sensing, has attracted considerable interest due to its importance in ubiquitous monitoring of cardiopulmonary health- and sleep-related biometrics. However, the sensitivity and dynamic range of prevalent sensors are often insufficient to detect subtle body signals. This study introduces a novel capacitive nanocomposite proximity-pressure sensor (NPPS) for detecting multiple human biometrics. NPPS consists of a carbon nanotube-paper composite (CPC) electrode and a percolating multiwalled carbon nanotube (MWCNT) foam enclosed in a MWCNT-coated auxetic frame. The fractured fibers in the CPC electrode intensify an electric field, enabling highly sensitive detection of proximity and pressure. When pressure is applied to the sensor, the synergic effect of MWCNT foam and auxetic deformation amplifies the sensitivity. The simple and mass-producible fabrication protocol allows for building an array of highly sensitive sensors to monitor human presence, sleep posture, and vital signs, including ballistocardiography (BCG). With the aid of a machine learning algorithm, the sensor array accurately detects blood pressure (BP) without intervention. This advancement holds promise for unrestricted vital sign monitoring during sleep or driving.

AL-MobileNet: a novel model for 2D gesture recognition in intelligent cockpit based on multi-modal data

As the degree of automotive intelligence increases, gesture recognition is gaining more attention in human-vehicle interaction. However, existing gesture recognition methods are computationally intensive and perform poorly in multi-modal sensor scenarios. This paper proposes a novel network structure, AL-MobileNet (MobileNet with Attention and Lightweight Modules), which can quickly and accurately estimate 2D gestures in RGB and infrared (IR) images. The innovations of this paper are as follows: Firstly, to enhance multi-modal data, we created a synthetic IR dataset based on real 2D gestures and employed a coarse-to-fine training approach. Secondly, to speed up the model's computation on edge devices, we introduced a new lightweight computational module called the Split Channel Attention Block (SCAB). Thirdly, to ensure the model maintains accuracy in large datasets, we incorporated auxiliary networks and Angle-Weighted Loss (AWL) into the backbone network. Experiments show that AL-MobileNet requires only 0.4 GFLOPs of computational power and 1.2 million parameters. This makes it 1.5 times faster than MobileNet and allows for quick execution on edge devices. AL-MobileNet achieved a running speed of up to 28 FPS on the Ambarella CV28. On both general datasets and our dataset, our algorithm achieved an average PCK0.2 score of 0.95. This indicates that the algorithm can quickly generate accurate 2D gestures. The demonstration of the algorithm can be reviewed in gesturebaolong.

Molecularly Imprinted Wearable Sensor with Paper Microfluidics for Real-Time Sweat Biomarker Analysis.

The urgent need for real-time and noninvasive monitoring of health-associated biochemical parameters has motivated the development of wearable sweat sensors. Existing electrochemical sensors show promise in real-time analysis of various chemical biomarkers. These sensors often rely on labels and redox probes to generate and amplify the signals for the detection and quantification of analytes with limited sensitivity. In this study, we introduce a molecularly imprinted polymer (MIP)-based biochemical sensor to quantify a molecular biomarker in sweat using electrochemical impedance spectroscopy, which eliminates the need for labels or redox probes. The molecularly imprinted biosensor can achieve sensitive and specific detection of cortisol at concentrations as low as 1 pM, 1000-fold lower than previously reported MIP cortisol sensors. We integrated multimodal electrochemical sensors with an iontophoresis sweat extraction module and paper microfluidics for real-time sweat analysis. Several parameters can be simultaneously quantified, including sweat volume, secretion rate, sodium ion, and cortisol concentration. Paper microfluidic modules not only quantify sweat volume and secretion rate but also facilitate continuous sweat analysis without user intervention. While we focus on cortisol sensing as a proof-of-concept, the molecularly imprinted wearable sensors can be extended to real-time detection of other biochemicals, such as protein biomarkers and therapeutic drugs.

A Battery‐Free Wireless Tactile Sensor for Multimodal Force Perception

AbstractMultimodal tactile sensors, as key information input channel in human‐machine interactions, have faced the significant challenges including high power‐consumption, multimodal data fusion, and wireless transmission. In this work, a battery‐free multimodal wireless tactile sensor (TC‐MWTS) based on tribo‐capacitive coupled effect for normal and shear force fusion sensing is proposed, which is enabled by a 3D structure combining a triboelectric sensor and a capacitive sensor coupled with an inductive coil. A triboelectric sensor equipped with contact‐discharge structures exhibits 25‐fold wireless signal enhancement compared to conventional triboelectric sensors. Based on the characteristics of dual time‐frequency domain information existing in the wireless signals, both normal and shear forces can simultaneously be converted into voltage amplitude V and eigenfrequency f, respectively, without crosstalk and complex decoupling signals. The TC‐MWTS exhibits a maximum sensitivity of 2.47 V kPa−1 for normal force from 2 to 30 kPa and a sensitivity of 0.28 MHz N−1 for shear force between 0.3 and 1.0 N. Finally, the excellent sensing capability of TC‐MWTS to sense complex multidimensional forces in human‐machine interaction is demonstrated. This work innovatively proposes a new mechanism and methodology for effectively fusing and processing multimodal tactile information, which may drive the tremendous development of low‐power multimodal tactile sensing system.

Critical infrastructure monitoring in CBRNe scenarios: a reliable and robust communication network for distributed multimodal sensors

The malicious misuse of CBRNe agents can inflict extensive damage to critical infrastructure and terrorize public society. The effects of such attacks can range from financial or structural drawbacks to significant casualties. Recent terror attacks in Cucuta (2021) or Kabul (2021) demonstrate the need to protect infrastructures such as airports, power plants, and transportation infrastructure. In the face of these modern-day threats, the need for a reliable and effective monitoring system for critical infrastructure has become increasingly important. In a first contribution to the long-term development, we present an initial version of a concept that requires the implementation paradigm of diverse redundancy and secure communication using MQTTS relying on the included cipher suite. An initial set of node types is defined and assigned requirements for implementation. The software tools architecture we propose based on these requirements is designed to support network operators and developers by providing standard features for network management and sensor node implementation. This includes the definition of a secure semi-automatic onboarding process for new sensor nodes, which is presented in detail. We strive for independency from specific hardware platforms, software frameworks, and network technologies to enable an open standard for communication within a critical infrastructure sensor network and also between such networks in the future.

Correlations between Social Isolation and Functional Decline in Older Adults after Lower Limb Fractures Using Multimodal Sensors: A Pilot Study

Older adults (OAs) recovering from lower limb fractures experience social isolation (SI) and functional decline (FD) after they are discharged from inpatient rehabilitation due to reduced physical mobility. Our research used MAISON (Multimodal AI-based Sensor platform for Older iNdividuals), a multimodal sensor system comprising various smart devices collecting acceleration, heart rate, step count, frequency of indoor motion, GPS, and sleep metrics. This study aimed to investigate the correlations between SI and FD with multimodal sensor data from OAs following lower limb fractures. Multimodal sensor data from eight OAs (8 weeks per person) living at home were collected. Five clinical metrics were obtained via biweekly video calls, including three clinical questionnaires (Social Isolation Scale (SIS), Oxford Hip Score, Oxford Knee score) and two physical mobility assessments (Timed Up and Go, 30 s chair stand). From the sensor data collected, 53 statistical and domain features were extracted. Spearman correlation coefficients were calculated between the extracted features and clinical data. The results indicated strong correlations between various items of SIS and sleep metrics in OAs and various items of Oxford Knee Score with GPS and acceleration data. Strong correlations between the questions of the Oxford scores and sensor data highlight the direct impact of physical health status on measurable daily physical activities.

Joint Domain Adaptation-Based Lightweight Approach for Cross-Domain Diagnosis Compatible With Different Devices and Multimodal Sensing

Joint Domain Adaptation-Based Lightweight Approach for Cross-Domain Diagnosis Compatible With Different Devices and Multimodal Sensing

A Wireless Signal Correlation Learning Framework for Accurate and Robust Multi-Modal Sensing

A Wireless Signal Correlation Learning Framework for Accurate and Robust Multi-Modal Sensing

Building covalent crosslinks of carboxymethyl konjac glucomannan with boronic ester bonds for fabricating multimodal hydrogel sensor

As the derivative of konjac glucomannan (KGM), carboxymethyl konjac glucomannan (CMK) has attracted increasing attention in the polysaccharide hydrogel fields with the aim of improving the performance related to drug delivery and release. In this study, we prepared a CMK-based hydrogel with dual characteristic crosslinks, and unlocked new applications of this type of hydrogel in soft sensor fields. CMK and poly (vinyl alcohol) were used as substrates, and physical crosslinks were constructed via the freeze-thawing treatments and covalent crosslinks were built via the boronic ester bonding. As-prepared hydrogel possessed significantly improved mechanical performance because the boronic ester bonding, on the one hand, well associated the two kinds of polymer chains, and on the other hand, played the role of ‘sacrificial crosslinks’. Furthermore, the occurrence of dynamic boronic ester bonding gave the hydrogel strain- and temperature-sensitive ionic conductivity, and therefore, the hydrogels could be used to identify human motions and as-resulted environmental temperature alterations, and worked well in various scenarios. This work activates new application of CMK in the multimodal sensing field, and also proposes an intriguing way of building multiple crosslinks in the KGM derivative-based hydrogels.

Multivalent Pt/Ti3C2Tx Nanocomposite-Based Immunochromatographic Sensor for Colorimetric/Temperature/Pressure Trimodal Detection.

Multimodal immunochromatographic sensors (ICSs) have acquired extensive attention since they not only provide reliable results by comparing the different output signals but also flexibly respond to various application environments. Herein, an ICS with triple signal outputs including colorimetry, temperature, and pressure was developed for sensitive detection of chlorothalonil. The multivalent Pt/Ti3C2Tx nanoparticles as signal tags were facilely synthesized by loading PtNPs onto single-layer Ti3C2Tx nanosheets with high surface area. The acquired Pt/Ti3C2TxNPs accelerated the rate-limiting step of the aerogenesis reaction of H2O2 for producing intensive pressure signals due to their significant catalase-mimic activity. Meanwhile, they showed desirable photothermal conversion efficiency in the near-infrared region for producing significant temperature signals. Furthermore, their deep color also allowed facile colorimetry by using the naked eye. Based on a competitive immunoassay, chlorothalonil was detected as a model analyte on this trimodal ICS platform. The detection limits for pressure, temperature, and colorimetric modes were 0.04, 0.09, and 5 ng mL-1, respectively. The recoveries for detecting chlorothalonil supplemented in Astragalus and Honeysuckle with pressure mode were 84.0-110% and 108-114%, respectively. Therefore, the ICS presented a portable, sensitive, accurate, and flexible multimodal strategy suitable for point-of-care testing.