Multimodal Sensor Research Articles

Multichannel multimodal piezoelectric middle ear implant concept based on MEMS technology for next-generation fully implantable cochlear implant applications

This paper introduces a unique multimode, multichannel piezoelectric vibration sensor for the next-generation fully implantable cochlear implant (FICI) systems. The sensor, which can be implanted on the middle ear chain to collect and filter the ambient sound in eight frequency bands, comprises an array of 4 M-shape multimode and 11 single cantilevers. Finite element (FE) analysis indicates a 2.05-fold improvement in capturing frequency information for the multimodal sensor compared to its single-mode counterpart. Under an acoustic excitation at 100 dB SPL, the sensor, mounted on an artificial tympanic membrane, yielded a peak output voltage of 546.16 mVpp and a peak sensitivity of 285.28 mVpp/Pa at 1613 Hz. The extrapolated acoustic results indicated a dynamic frequency range between 300 Hz and 6 kHz, even at 30 dB SPL. Furthermore, a lightweight titanium coupler, employing a two-sided clipping structure with a maximum wall thickness of 70 μm, is micromachined for surgical attachment of the transducer to the middle ear chain. A commercial accelerometer, implanted on the incus short process (SP) of a cadaver using the titanium coupler, successfully recorded 0.1 g for 100 dB SPL at 500 Hz, revealing the potential feasibility of the coupler for vibration sensor implantation. Moreover, the presented anatomically accurate FE model of the middle ear, exhibiting a high correlation coefficient (R2) of 0.97 with the cadaveric experiment, suggests an efficient numerical approach for evaluating the implantation of middle ear prostheses. In this regard, the study holds great promise for clinical application in the field of implantable hearing aids.

Fire Detection in Urban Areas Using Multimodal Data and Federated Learning

Fire chemical sensing for indoor detection of fire plays an essential role because it can detect chemical volatiles before smoke particles, providing a faster and more reliable method for early fire detection. A thermal imaging camera and seven distinct fire-detecting sensors were used simultaneously to acquire the multimodal fire data that is the subject of this paper. The low-cost sensors typically have lower sensitivity and reliability, making it impossible for them to detect fire at greater distances. To go beyond the limitation of using solely sensors for identifying fire, the multimodal dataset is collected using a thermal camera that can detect temperature changes. The proposed pipeline uses image data from thermal cameras to train convolutional neural networks (CNNs) and their many versions. The training of sensors data (from fire sensors) uses bidirectional long-short memory (BiLSTM-Dense) and dense and long-short memory (LSTM-DenseDenseNet201), and the merging of both datasets demonstrates the performance of multimodal data. Researchers and system developers can use the dataset to create and hone cutting-edge artificial intelligence models and systems. Initial evaluation of the image dataset has shown densenet201 as the best approach with the highest validation parameters (0.99, 0.99, 0.99, and 0.08), i.e., Accuracy, Precision, Recall, and Loss, respectively. However, the sensors dataset has also shown the highest parameters with the BILSTM-Dense approach (0.95, 0.95, 0.95, 0.14). In a multimodal data approach, image and sensors deployed with a multimodal algorithm (densenet201 for image data and Bi LSTM- Dense for Sensors Data) has shown other parameters (1.0, 1.0, 1.0, 0.06). This work demonstrates that, in comparison to the conventional deep learning approach, the federated learning (FL) approach performs privacy-protected fire leakage classification without significantly sacrificing accuracy and other validation parameters.

A Multimodal Bio‐Sensing and Imaging Platform (MSIP) for Real‐Time Intraoperative Margin Detection in Cancer Surgery

AbstractIn cancer surgery, the precise characterisation of tissue margins is of paramount importance to ensure effective resection. However, conventional methods have inherent limitations in terms of both speed and accuracy, often necessitating labour‐intensive postsurgical tissue analyses. To overcome these challenges and enhance the precision of cancer surgery, we introduced an innovative Multimodal Sensing and Imaging Platform (MSIP) equipped with advanced pH sensing capabilities. The MSIP represents the synergistic fusion of a miniaturised, flexible wire‐based electrochemical pH sensor with a sensitivity of 62.4 mV per pH for real‐time functional imaging. Simultaneously, it incorporates a high‐speed confocal endomicroscope utilizing a 2.6 mm fibre bundle to achieve subcellular resolution in morphological imaging. The pH sensor enables continuous monitoring of dynamic changes in the tumour microenvironment, offering valuable insights. Confocal endomicroscopy provides high‐resolution, real‐time imaging of tumour morphology with an impressive 2.2 μm resolution. This integration not only enhances the precision of cancer surgery but also fosters a comprehensive understanding of the biological and molecular characteristics of tumours. By consolidating multiple diagnostic parameters into a unified platform, MSIP has emerged as a transformative technology poised to revolutionise our approach to cancer surgery.

A Curb-Detection Network with a Tri-Plane BEV Encoder Module for Autonomous Delivery Vehicles

Curb detection tasks play a crucial role in the perception of the autonomous driving environment for logistics vehicles. With the popularity of multi-modal sensors under the BEV (Bird’s Eye View) paradigm, curb detection tasks are increasingly being integrated into multi-task perception networks, achieving robust detection results. This paper modifies and integrates the tri-plane spatial feature representation method of the EG3D network from the field of 3D reconstruction into a BEV-based multi-modal sensor detection network, including LiDAR, pinhole cameras, and fisheye cameras. The system collects a total of 24,350 frames of data under real road conditions for experimentation, proving the effectiveness of the proposed method.

Design of an optical sensor based on organic optoelectronics and nanoplasmonics for multiplex and multimodal detection

—On-site analysis of multiple analytes from different classes (such as heavy metals, proteins and small molecules), at the sensitivity required for a selected application, is a hard technological challenge. In this context, optical sensing in miniaturized systems has the largest potential. We present here the design and optimization of a miniaturized optical sensor with multiple channels, capable of multimodal optical detection in each channel, and the proof-of-concept realization of sub-systems providing two complementary detection modes: plasmon enhanced fluorescence and localized surface plasmon resonance. The multichannel (enabling multiplexing) and multimodal optical sensor is designed to have a total size of one inch-square and optimized sensing performance, obtained by combining organic optoelectronic and nanoplasmonic components.

Multimodal Daily-Life Logging in Free-living Environment Using Non-Visual Egocentric Sensors on a Smartphone

Egocentric non-intrusive sensing of human activities of daily living (ADL) in free-living environments represents a holy grail in ubiquitous computing. Existing approaches, such as egocentric vision and wearable motion sensors, either can be intrusive or have limitations in capturing non-ambulatory actions. To address these challenges, we propose EgoADL, the first egocentric ADL sensing system that uses an in-pocket smartphone as a multi-modal sensor hub to capture body motion, interactions with the physical environment and daily objects using non-visual sensors (audio, wireless sensing, and motion sensors). We collected a 120-hour multimodal dataset and annotated 20-hour data into 221 ADL, 70 object interactions, and 91 actions. EgoADL proposes multi-modal frame-wise slow-fast encoders to learn the feature representation of multi-sensory data that characterizes the complementary advantages of different modalities and adapt a transformer-based sequence-to-sequence model to decode the time-series sensor signals into a sequence of words that represent ADL. In addition, we introduce a self-supervised learning framework that extracts intrinsic supervisory signals from the multi-modal sensing data to overcome the lack of labeling data and achieve better generalization and extensibility. Our experiments in free-living environments demonstrate that EgoADL can achieve comparable performance with video-based approaches, bringing the vision of ambient intelligence closer to reality.

Breath analysis system with convolutional neural network (CNN) for early detection of lung cancer

Early diagnosis of lung cancer, the leading cause of cancer-related death worldwide, is critical for reducing mortality rate. However, current diagnostic methods are invasive, time-consuming, costly, and may not always provide accurate diagnoses. For early diagnosis, recent research has focused on noninvasive approaches, including the detection of volatile organic compounds (VOCs) in human exhaled breath. Changes in the composition and concentration of VOCs in exhaled breath may indicate lung cancer, and this approach offers several advantages over traditional diagnostic methods. Moreover, the combination of a breath gas sensing system and machine learning algorithms provides a more accurate diagnosis. In this study, for the early diagnosis of lung cancer, a breath analysis system was developed using a gas sensor array and deep learning algorithm. The breath analysis system was designed to detect multiple VOCs in exhaled breath using ten semiconductor metal oxide (SMO), one photoionization detector (PID), nine electrochemical (EC) gas sensors. In total, 181 clinical breath samples (from 74 healthy controls and 107 lung cancer patients) were collected and analyzed using a 1D convolutional neural network (CNN) algorithm. The results showed an overall accuracy of 97.8% in classifying healthy controls and lung cancer patients using a complete clinical dataset. Through a comparison of the single-sensor type data and multimodal sensor data and performance analysis of three different deep learning models (multilayer perceptron, recurrent neural network, and CNN), we validated the potential of the breath analyzer with a multimodal sensor system and a 1D CNN as a lung cancer diagnostic device.

Granular estimation of user cognitive workload using multi-modal physiological sensors.

Mental workload (MWL) is a crucial area of study due to its significant influence on task performance and potential for significant operator error. However, measuring MWL presents challenges, as it is a multi-dimensional construct. Previous research on MWL models has focused on differentiating between two to three levels. Nonetheless, tasks can vary widely in their complexity, and little is known about how subtle variations in task difficulty influence workload indicators. To address this, we conducted an experiment inducing MWL in up to 5 levels, hypothesizing that our multi-modal metrics would be able to distinguish between each MWL stage. We measured the induced workload using task performance, subjective assessment, and physiological metrics. Our simulated task was designed to induce diverse MWL degrees, including five different math and three different verbal tiers. Our findings indicate that all investigated metrics successfully differentiated between various MWL levels induced by different tiers of math problems. Notably, performance metrics emerged as the most effective assessment, being the only metric capable of distinguishing all the levels. Some limitations were observed in the granularity of subjective and physiological metrics. Specifically, the subjective overall mental workload couldn't distinguish lower levels of workload, while all physiological metrics could detect a shift from lower to higher levels, but did not distinguish between workload tiers at the higher or lower ends of the scale (e.g., between the easy and the easy-medium tiers). Despite these limitations, each pair of levels was effectively differentiated by one or more metrics. This suggests a promising avenue for future research, exploring the integration or combination of multiple metrics. The findings suggest that subtle differences in workload levels may be distinguishable using combinations of subjective and physiological metrics.

Ultra‐Stretchable and Environmentally Resilient Hydrogels Via Sugaring‐Out Strategy for Soft Robotics Sensing

AbstractThe adoption of hydrogels in most applications is hampered by their high free water content, which limits their mechanical performance and environmental resilience. Herein, this issue is simultaneously addressed by modulating the state of water and the intermolecular interactions in polyacrylamide (PAM) hydrogels. Specifically, PAM hydrogels are toughened by sugaring‐out using a monosaccharide (glucose, G). Glucose is found to facilitate PAM hydrogen bonding and interchain interactions. Meanwhile, the high hygroscopicity of glucose converts some of the free water to bound state, endowing the hydrogels with remarkable resilience to extreme environmental conditions. The PAM‐G hydrogels are demonstrated as multimodal sensors for soft robotics. Moreover, PAM‐G alcogels produced by solvent exchanging with ethanol are shown as effective opto‐mechanical sensors. Notably, all these properties are obtained by the inclusion of glucose, a green additive showing no negative health and environmental effect.

Flexible Multimodal Sensing System Based on a Vertical Stacking Strategy for Efficiently Decoupling Multiple Signals.

Multisensory integration enables the simultaneous perception of multiple environmental stimuli while minimizing size and energy consumption. However, conventional multifunctional integration in flexible electronics typically requires large-scale horizontal sensing arrays (such as flexible printed circuit boards), posing decoupling complexities, tensile strain limitation, and spatial constraints. Herein, a fully flexible multimodal sensing system (FMSS) is developed by coupling biomimetic stretchable conductive films (BSCFs) and strain-insensitive communication interfaces using a vertical stacking integration strategy. The FMSS achieves vertical integration without additional adhesives, and it can incorporate individual sensing layers and stretchable interconnects without any essential constraint on their deformations. Accordingly, the temperature and pressure are precisely decoupled simultaneously, and tensile stress can be accurately discerned in different directions. This vertical stacking integration strategy is expected to offer a new approach to significantly streamline the design and fabrication of multimodal sensing systems and enhance their decoupling capabilities.

Developing a Smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors Integration and Internet of Things

Developing a Smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors Integration and Internet of Things

Multimodal Sensors Enabled Autonomous Soft Robotic System with Self-Adaptive Manipulation.

Human hands are amazingly skilled at recognizing and handling objects of different sizes and shapes. To date, soft robots rarely demonstrate autonomy equivalent to that of humans for fine perception and dexterous operation. Here, an intelligent soft robotic system with autonomous operation and multimodal perception ability is developed by integrating capacitive sensors with triboelectric sensor. With distributed multiple sensors, our robot system can not only sense and memorize multimodal information but also enable an adaptive grasping method for robotic positioning and grasp control, during which the multimodal sensory information can be captured sensitively and fused at feature level for crossmodally recognizing objects, leading to a highly enhanced recognition capability. The proposed system, combining the performance and physical intelligence of biological systems (i.e., self-adaptive behavior and multimodal perception), will greatly advance the integration of soft actuators and robotics in many fields.

Exploring deep learning techniques for wild animal behaviour classification using animal‐borne accelerometers

Abstract Machine learning‐based behaviour classification using acceleration data is a powerful tool in bio‐logging research. Deep learning architectures such as convolutional neural networks (CNN), long short‐term memory (LSTM) and self‐attention mechanism as well as related training techniques have been extensively studied in human activity recognition. However, they have rarely been used in wild animal studies. The main challenges of acceleration‐based wild animal behaviour classification include data shortages, class imbalance problems, various types of noise in data due to differences in individual behaviour and where the loggers were attached and complexity in data due to complex animal‐specific behaviours, which may have limited the application of deep learning techniques in this area. To overcome these challenges, we explored the effectiveness of techniques for efficient model training: data augmentation, manifold mixup and pre‐training of deep learning models with unlabelled data, using datasets from two species of wild seabirds and state‐of‐the‐art deep learning model architectures. Data augmentation improved the overall model performance when one of the various techniques (none, scaling, jittering, permutation, time‐warping and rotation) was randomly applied to each data during mini‐batch training. Manifold mixup also improved model performance, but not as much as random data augmentation. Pre‐training with unlabelled data did not improve model performance. The state‐of‐the‐art deep learning models, including a model consisting of four CNN layers, an LSTM layer and a multi‐head attention layer, as well as its modified version with shortcut connection, showed better performance among other comparative models. Using only raw acceleration data as inputs, these models outperformed classic machine learning approaches that used 119 handcrafted features. Our experiments showed that deep learning techniques are promising for acceleration‐based behaviour classification of wild animals and highlighted some challenges (e.g. effective use of unlabelled data). There is scope for greater exploration of deep learning techniques in wild animal studies (e.g. advanced data augmentation, multimodal sensor data use, transfer learning and self‐supervised learning). We hope that this study will stimulate the development of deep learning techniques for wild animal behaviour classification using time‐series sensor data.

Validating EmotiBit, an open-source multi-modal sensor for capturing research-grade physiological signals from anywhere on the body

Peripheral physiological signals provide a powerful window into understanding the body and mind. Peripheral physiology devices that are currently available on the market, however, face a number of challenges. Consumer-grade devices (e.g. FitBit, Apple Watch) are easy to wear, but provide limited access to unprocessed data. Combined with black-box signal processing algorithms, this makes it difficult to interpret the data for scientific research purposes. Research-grade devices (e.g. Empatica, Shimmer, BIOPAC) provide greater access to high-quality data but remain in closed ecosystems and at price points that are out of reach for many. To bridge the gaps in available biometric solutions, our labs have created an open-source physiological sensing platform called EmotiBit (http://www.emotibit.com/), measuring EDA, multi-wavelength PPG, temperature, and 9-axis IMU. This study compares EmotiBit biometric signals to gold-standard devices by Brain Products and finds that the physiological signals exhibit a high degree of similarity, validating their use in research.

CrossPrune: Cooperative pruning for camera–LiDAR fused perception models of autonomous driving

Deep neural network pruning is effective in enabling high-performance perception models to be deployed on autonomous driving platforms with limited computation and memory resources. With their rapid development, state-of-the-art autonomous driving perception (ADP) models have advocated the use of multimodal sensors to extract diverse feature categories. However, existing pruning studies in the ADP area focus on single-modal models and neglect multimodal models. Compared with conventional pruning, multimodal pruning presents a new type of redundancy, namely modal-wise redundancy that is caused by multimodal branches extracting similar perception information. When a specific type of information is extracted by more than one modal branch, although this extraction is deemed essential from an individual single-modal standpoint, it becomes redundant when viewed from a modal-wise perspective. Therefore, modal branches must be handled cooperatively to eliminate modal-wise redundancy while concurrently preserving the original perception accuracy. Building on this, we propose CrossPrune, a modal cooperative pruning framework designed for camera–LiDAR fused perception in autonomous driving. The primary objective of CrossPrune is to effectively eliminate redundancy and achieve nondestructive pruning for multimodal ADP models. This was accomplished by approaching the problem as a multi-objective optimization task, encompassing both weight pruning and the restriction of feature distortions caused by pruning. Experiments conducted on the nuScenes and KITTI datasets demonstrated that CrossPrune attained superior pruning ratios while minimizing accuracy loss, surpassing the performance of the baselines. The key results indicated that the proposed CrossPrune achieved relative improvements of 9.6% in mAP and 11.5% in NDS under 89.8% pruning sparsity.

Using bacterial cellulose to bridge covalent and physical crosslinks in hydrogels for fabricating multimodal sensors

Network optimization is vital for the polysaccharide based hydrogels with multiple crosslinks. In this study, we developed a ‘two-step’ strategy to activate synergistic effect of chemical and physical crosslinks using a poly (vinyl alcohol) (PVA)/bacterial cellulose (BC) hydrogel as a template. The BC nanofibers, on the one hand, acted as nucleating agents, participating in the crystallization of PVA, and on the other hand, were also involved in the formation of boronic ester bond, anchored with the PVA chains via chemical bonding. Therefore, the existence of BC nanofibers, as ‘bridge’, linked the crystalline regions and amorphous parts of PVA together, associating the two characteristic crosslinks, which was conducive to load transfer. The mechanical properties of resultant hydrogels, including the tensile elongation and strength, as well as fracture toughness, were significantly improved. Moreover, the dually cross-linked hydrogels possessed ionic conductivity, which was sensitive to the tensile deformation and environmental temperature. This study clarifies a unique role of BC nanofibers in hydrogels, and proposes an effective approach to construct multiple networks in the nanocellulose reinforced PVA hydrogels.

Adaptive low-temperature plasmas

This paper highlights the uniqueness of low-temperature plasma (LTP), that is its ability to change the chemical composition in situ. An adaptive LTP platform is a plasma device that can adjust the plasma composition to obtain optimal desirable outcomes through its interaction with a target. This approach relies on the ability of measuring the real time response of target to plasma action. In biological applications, feedback consists of the cellular response to plasma immediately after treatment and modifying the composition and power of plasma via a feedback mechanism. Plasma self-adaptation might be feasible due to self-organization and pattern formation when plasma interacts with targets. An adaptive plasma system requires an integration with multi-modal sensors, augmented by artificial intelligence where appropriate. Such adaptive LTP platforms might lead to treatment of biological targets on demand at a dose that is optimized by outcome (e.g. decontamination, disinfection) at the lowest possible cost and smallest environmental impact. In this paper, we explore the possibilities and opportunities for using the adaptive plasma as a therapeutic system.

TacSuit: A Wearable Large-Area, Bioinspired Multimodal Tactile Skin for Collaborative Robots

Robots are now working more and more closely with humans outside of traditional fences in industrial scenes. Their real-time tactile interaction perception is crucial to the safety of human–robot collaboration (HRC). In this work, we present a customized, wearable and modular robot skin (TacSuit), which is scalable for large-area surface coverage of robot with easily accessible multi-modal sensors, including pressure, proximity, acceleration and temperature sensors. The TacSuit is co-designed for mechanical structure and data fusion algorithm, consisting of three levels of design: sensor, cell (of multi-modal sensors) and block (of multiple cells). These sensors are stored with custom-designed and 3D printed capsules to achieve the conformity, scalability and easy installation to the arbitrary robot surface. A multi-level event-driven data fusion algorithm enables efficient information processing for large number of tactile sensors. Furthermore, a virtual interaction force fusion method takes both the proximity and force perception information into consideration in order to achieve safety of whole interaction process before and after direct physical contacts. A humanoid robotic platform successfully realizes the TacSuit wear of 159 tactile cells. Validation experiments of obstacle detection demonstrate the effective collision avoidance capability of the TacSuit for safe HRC.

Machine learning-based prediction of cerebral oxygen saturation based on multi-modal cerebral oximetry data.

This study develops machine learning-based algorithms that facilitate accurate prediction of cerebral oxygen saturation using waveform data in the near-infrared range from a multi-modal oxygen saturation sensor. Data were obtained from 150,000 observations of a popular cerebral oximeter, Masimo O3™ regional oximetry (Co., United States) and a multi-modal cerebral oximeter, Votem (Inc., Korea). Among these observations, 112,500 (75%) and 37,500 (25%) were used for training and test sets, respectively. The dependent variable was the cerebral oxygen saturation value from the Masimo O3™ (0-100%). The independent variables were the time of measurement (0-300,000 ms) and the 16-bit decimal amplitudes values (infrared and red) from Votem (0-65,535). For the right part of the forehead, the root mean square error of the random forest (0.06) was much smaller than those of linear regression (1.22) and the artificial neural network with one, two or three hidden layers (2.58). The result was similar for the left part of forehead, that is, random forest (0.05) vs logistic regression (1.22) and the artificial neural network with one, two or three hidden layers (2.97). Machine learning aids in accurately predicting of cerebral oxygen saturation, employing the data from a multi-modal cerebral oximeter.

Use of voice features from smartphones for monitoring depressive disorders: Scoping review.

This review evaluates the use of smartphone-based voice data for predicting and monitoring depression. A scoping review was conducted, examining 14 studies from Medline, Scopus, and Web of Science (2010-2023) on voice data collection methods and the use of voice features for minitoring depression. Voice data, especially prosodic features like fundamental frequency and pitch, show promise for predicting depression, though their sole predictive power requires further validation. Integrating voice with multimodal sensor data has been shown to improve accuracy significantly. Smartphone-based voice monitoring offers a promising, noninvasive, and cost-effective approach to depression management. The integration of machine learning with sensor data could significantly enhance mental health monitoring, necessitating further research and longitudinal studies for validation.