Multimodal Sensor Research Articles

L-NORM: Learning and Network Orchestration at the Edge for Robot Connectivity and Mobility in Factory Floor Environments

Robotic factory floors will revolutionize the future of manufacturing and the service industry by automating tasks. However, to fully supplement human effort, these robots will need low-latency, reliable connectivity throughout the work zone through links established by wireless access points (APs). This will allow the robot to assuredly respond to programming directives that rely on the real-time relaying of robot-generated sensor data to the Mobile Edge Computing (MEC) server. In this paper, we propose L-NORM, a multi-AP and multi-robot coordination framework, as a multi-tiered solution for such autonomous edge networks. First, multi-robot motion planning through reinforcement learning occurs at the MEC, using as input multi-modal robot sensor data. Second, multi-AP resource orchestration is performed using another reinforcement learning-based method that maps a subset of available APs to each robot toward meeting their sensor data delivery requirements. Furthermore, we suggest diversity combination of uplink channels with the 802.11ax scheduled access mode that will (i) support high reliability of multi-robot uplink sensor packets and (ii) enable multi-AP coordination, for optimized resource utilization. Through extensive simulation studies, we show that the probability of robot deviation to remain within 0.5 m from its optimal path, is 19% more in L-NORM compared to classical 802.11ax based edge network solution, considering <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 1 MB of sensor data per robot.

Unified Adversarial Patch for Visible-Infrared Cross-Modal Attacks in the Physical World.

Physical adversarial attacks have put a severe threat to DNN-based object detectors. To enhance security, a combination of visible and infrared sensors is deployed in various scenarios, which has proven effective in disabling existing single-modal physical attacks. To further demonstrate the potential risks in such cases, we design a unified adversarial patch that can perform cross-modal physical attacks, achieving evasion in both modalities simultaneously with a single patch. Given the different imaging mechanisms of visible and infrared sensors, our work manipulates patches' shape features, which can be captured in different modalities when they undergo changes. To deal with challenges, we propose a novel boundary-limited shape optimization approach that aims to achieve compact and smooth shapes for the adversarial patch, making it easy to implement in the physical world. And a score-aware iterative evaluation method is also introduced to balance the fooling degree between visible and infrared detectors during optimization, which guides the adversarial patch to iteratively reduce the predicted scores of the multi-modal sensors. Furthermore, we propose an Affine-Transformation-based enhancement strategy that makes the learnable shape robust to various angles, thus mitigating the issue of shape deformation caused by different shooting angles in the real world. Our method is evaluated against several state-of-the-art object detectors, achieving an Attack Success Rate (ASR) of over 80%. We also demonstrate the effectiveness of our approach in physical-world scenarios under various settings, including different angles, distances, postures, and scenes for both visible and infrared sensors.

AWLC: Adaptive Weighted Loop Closure for SLAM with Multi-Modal Sensor Fusion

The present prevailing loop closure detection algorithm is mainly applicable for simultaneous localization and mapping (SLAM). Its effectiveness is contingent upon environmental conditions, which can fluctuate due to variations in lighting or the surrounding scenario. Vision-based algorithms, while adept during daylight hours, may falter in nocturnal settings. Conversely, lidar methods hinge on the sparsity of the given scenario. This paper proposes an algorithm that comprehensively utilizes lidar and image features to assign weighted factors for loop closure detection based on multi-modal sensor fusion. First, we use [Formula: see text]-means clustering to produce a point cloud spatial global bag of words. Second, an improved deep learning method is used to train feature descriptors of images while scan context is also used to detect candidate point cloud features. After that, different feature-weighted factors are assigned for homologous feature descriptors. Finally, the detection result related to the maximum weight factor is designated to the optimal loop closure. The adaptive weighted loop closure (AWLC) algorithm we proposed inherits the advantages of different loop closure detection algorithms and hence it is accurate and robust. The AWLC method is compared with popular loop detection algorithms in different datasets. Experiments show that the AWLC can maintain the effectiveness and robustness of detection even at night or in highly dynamic complex environment.

DistilVPR: Cross-Modal Knowledge Distillation for Visual Place Recognition

The utilization of multi-modal sensor data in visual place recognition (VPR) has demonstrated enhanced performance compared to single-modal counterparts. Nonetheless, integrating additional sensors comes with elevated costs and may not be feasible for systems that demand lightweight operation, thereby impacting the practical deployment of VPR. To address this issue, we resort to knowledge distillation, which empowers single-modal students to learn from cross-modal teachers without introducing additional sensors during inference. Despite the notable advancements achieved by current distillation approaches, the exploration of feature relationships remains an under-explored area. In order to tackle the challenge of cross-modal distillation in VPR, we present DistilVPR, a novel distillation pipeline for VPR. We propose leveraging feature relationships from multiple agents, including self-agents and cross-agents for teacher and student neural networks. Furthermore, we integrate various manifolds, characterized by different space curvatures for exploring feature relationships. This approach enhances the diversity of feature relationships, including Euclidean, spherical, and hyperbolic relationship modules, thereby enhancing the overall representational capacity. The experiments demonstrate that our proposed pipeline achieves state-of-the-art performance compared to other distillation baselines. We also conduct necessary ablation studies to show design effectiveness. The code is released at: https://github.com/sijieaaa/DistilVPR

Multichannel multimodal piezoelectric middle ear implant concept based on MEMS technology for next-generation fully implantable cochlear implant applications

This paper introduces a unique multimode, multichannel piezoelectric vibration sensor for the next-generation fully implantable cochlear implant (FICI) systems. The sensor, which can be implanted on the middle ear chain to collect and filter the ambient sound in eight frequency bands, comprises an array of 4 M-shape multimode and 11 single cantilevers. Finite element (FE) analysis indicates a 2.05-fold improvement in capturing frequency information for the multimodal sensor compared to its single-mode counterpart. Under an acoustic excitation at 100 dB SPL, the sensor, mounted on an artificial tympanic membrane, yielded a peak output voltage of 546.16 mVpp and a peak sensitivity of 285.28 mVpp/Pa at 1613 Hz. The extrapolated acoustic results indicated a dynamic frequency range between 300 Hz and 6 kHz, even at 30 dB SPL. Furthermore, a lightweight titanium coupler, employing a two-sided clipping structure with a maximum wall thickness of 70 μm, is micromachined for surgical attachment of the transducer to the middle ear chain. A commercial accelerometer, implanted on the incus short process (SP) of a cadaver using the titanium coupler, successfully recorded 0.1 g for 100 dB SPL at 500 Hz, revealing the potential feasibility of the coupler for vibration sensor implantation. Moreover, the presented anatomically accurate FE model of the middle ear, exhibiting a high correlation coefficient (R2) of 0.97 with the cadaveric experiment, suggests an efficient numerical approach for evaluating the implantation of middle ear prostheses. In this regard, the study holds great promise for clinical application in the field of implantable hearing aids.

Fire Detection in Urban Areas Using Multimodal Data and Federated Learning

Fire chemical sensing for indoor detection of fire plays an essential role because it can detect chemical volatiles before smoke particles, providing a faster and more reliable method for early fire detection. A thermal imaging camera and seven distinct fire-detecting sensors were used simultaneously to acquire the multimodal fire data that is the subject of this paper. The low-cost sensors typically have lower sensitivity and reliability, making it impossible for them to detect fire at greater distances. To go beyond the limitation of using solely sensors for identifying fire, the multimodal dataset is collected using a thermal camera that can detect temperature changes. The proposed pipeline uses image data from thermal cameras to train convolutional neural networks (CNNs) and their many versions. The training of sensors data (from fire sensors) uses bidirectional long-short memory (BiLSTM-Dense) and dense and long-short memory (LSTM-DenseDenseNet201), and the merging of both datasets demonstrates the performance of multimodal data. Researchers and system developers can use the dataset to create and hone cutting-edge artificial intelligence models and systems. Initial evaluation of the image dataset has shown densenet201 as the best approach with the highest validation parameters (0.99, 0.99, 0.99, and 0.08), i.e., Accuracy, Precision, Recall, and Loss, respectively. However, the sensors dataset has also shown the highest parameters with the BILSTM-Dense approach (0.95, 0.95, 0.95, 0.14). In a multimodal data approach, image and sensors deployed with a multimodal algorithm (densenet201 for image data and Bi LSTM- Dense for Sensors Data) has shown other parameters (1.0, 1.0, 1.0, 0.06). This work demonstrates that, in comparison to the conventional deep learning approach, the federated learning (FL) approach performs privacy-protected fire leakage classification without significantly sacrificing accuracy and other validation parameters.

A Multimodal Bio‐Sensing and Imaging Platform (MSIP) for Real‐Time Intraoperative Margin Detection in Cancer Surgery

AbstractIn cancer surgery, the precise characterisation of tissue margins is of paramount importance to ensure effective resection. However, conventional methods have inherent limitations in terms of both speed and accuracy, often necessitating labour‐intensive postsurgical tissue analyses. To overcome these challenges and enhance the precision of cancer surgery, we introduced an innovative Multimodal Sensing and Imaging Platform (MSIP) equipped with advanced pH sensing capabilities. The MSIP represents the synergistic fusion of a miniaturised, flexible wire‐based electrochemical pH sensor with a sensitivity of 62.4 mV per pH for real‐time functional imaging. Simultaneously, it incorporates a high‐speed confocal endomicroscope utilizing a 2.6 mm fibre bundle to achieve subcellular resolution in morphological imaging. The pH sensor enables continuous monitoring of dynamic changes in the tumour microenvironment, offering valuable insights. Confocal endomicroscopy provides high‐resolution, real‐time imaging of tumour morphology with an impressive 2.2 μm resolution. This integration not only enhances the precision of cancer surgery but also fosters a comprehensive understanding of the biological and molecular characteristics of tumours. By consolidating multiple diagnostic parameters into a unified platform, MSIP has emerged as a transformative technology poised to revolutionise our approach to cancer surgery.

A Curb-Detection Network with a Tri-Plane BEV Encoder Module for Autonomous Delivery Vehicles

Curb detection tasks play a crucial role in the perception of the autonomous driving environment for logistics vehicles. With the popularity of multi-modal sensors under the BEV (Bird’s Eye View) paradigm, curb detection tasks are increasingly being integrated into multi-task perception networks, achieving robust detection results. This paper modifies and integrates the tri-plane spatial feature representation method of the EG3D network from the field of 3D reconstruction into a BEV-based multi-modal sensor detection network, including LiDAR, pinhole cameras, and fisheye cameras. The system collects a total of 24,350 frames of data under real road conditions for experimentation, proving the effectiveness of the proposed method.

Design of an optical sensor based on organic optoelectronics and nanoplasmonics for multiplex and multimodal detection

—On-site analysis of multiple analytes from different classes (such as heavy metals, proteins and small molecules), at the sensitivity required for a selected application, is a hard technological challenge. In this context, optical sensing in miniaturized systems has the largest potential. We present here the design and optimization of a miniaturized optical sensor with multiple channels, capable of multimodal optical detection in each channel, and the proof-of-concept realization of sub-systems providing two complementary detection modes: plasmon enhanced fluorescence and localized surface plasmon resonance. The multichannel (enabling multiplexing) and multimodal optical sensor is designed to have a total size of one inch-square and optimized sensing performance, obtained by combining organic optoelectronic and nanoplasmonic components.

Multimodal Daily-Life Logging in Free-living Environment Using Non-Visual Egocentric Sensors on a Smartphone

Egocentric non-intrusive sensing of human activities of daily living (ADL) in free-living environments represents a holy grail in ubiquitous computing. Existing approaches, such as egocentric vision and wearable motion sensors, either can be intrusive or have limitations in capturing non-ambulatory actions. To address these challenges, we propose EgoADL, the first egocentric ADL sensing system that uses an in-pocket smartphone as a multi-modal sensor hub to capture body motion, interactions with the physical environment and daily objects using non-visual sensors (audio, wireless sensing, and motion sensors). We collected a 120-hour multimodal dataset and annotated 20-hour data into 221 ADL, 70 object interactions, and 91 actions. EgoADL proposes multi-modal frame-wise slow-fast encoders to learn the feature representation of multi-sensory data that characterizes the complementary advantages of different modalities and adapt a transformer-based sequence-to-sequence model to decode the time-series sensor signals into a sequence of words that represent ADL. In addition, we introduce a self-supervised learning framework that extracts intrinsic supervisory signals from the multi-modal sensing data to overcome the lack of labeling data and achieve better generalization and extensibility. Our experiments in free-living environments demonstrate that EgoADL can achieve comparable performance with video-based approaches, bringing the vision of ambient intelligence closer to reality.

Breath analysis system with convolutional neural network (CNN) for early detection of lung cancer

Early diagnosis of lung cancer, the leading cause of cancer-related death worldwide, is critical for reducing mortality rate. However, current diagnostic methods are invasive, time-consuming, costly, and may not always provide accurate diagnoses. For early diagnosis, recent research has focused on noninvasive approaches, including the detection of volatile organic compounds (VOCs) in human exhaled breath. Changes in the composition and concentration of VOCs in exhaled breath may indicate lung cancer, and this approach offers several advantages over traditional diagnostic methods. Moreover, the combination of a breath gas sensing system and machine learning algorithms provides a more accurate diagnosis. In this study, for the early diagnosis of lung cancer, a breath analysis system was developed using a gas sensor array and deep learning algorithm. The breath analysis system was designed to detect multiple VOCs in exhaled breath using ten semiconductor metal oxide (SMO), one photoionization detector (PID), nine electrochemical (EC) gas sensors. In total, 181 clinical breath samples (from 74 healthy controls and 107 lung cancer patients) were collected and analyzed using a 1D convolutional neural network (CNN) algorithm. The results showed an overall accuracy of 97.8% in classifying healthy controls and lung cancer patients using a complete clinical dataset. Through a comparison of the single-sensor type data and multimodal sensor data and performance analysis of three different deep learning models (multilayer perceptron, recurrent neural network, and CNN), we validated the potential of the breath analyzer with a multimodal sensor system and a 1D CNN as a lung cancer diagnostic device.

Granular estimation of user cognitive workload using multi-modal physiological sensors.

Mental workload (MWL) is a crucial area of study due to its significant influence on task performance and potential for significant operator error. However, measuring MWL presents challenges, as it is a multi-dimensional construct. Previous research on MWL models has focused on differentiating between two to three levels. Nonetheless, tasks can vary widely in their complexity, and little is known about how subtle variations in task difficulty influence workload indicators. To address this, we conducted an experiment inducing MWL in up to 5 levels, hypothesizing that our multi-modal metrics would be able to distinguish between each MWL stage. We measured the induced workload using task performance, subjective assessment, and physiological metrics. Our simulated task was designed to induce diverse MWL degrees, including five different math and three different verbal tiers. Our findings indicate that all investigated metrics successfully differentiated between various MWL levels induced by different tiers of math problems. Notably, performance metrics emerged as the most effective assessment, being the only metric capable of distinguishing all the levels. Some limitations were observed in the granularity of subjective and physiological metrics. Specifically, the subjective overall mental workload couldn't distinguish lower levels of workload, while all physiological metrics could detect a shift from lower to higher levels, but did not distinguish between workload tiers at the higher or lower ends of the scale (e.g., between the easy and the easy-medium tiers). Despite these limitations, each pair of levels was effectively differentiated by one or more metrics. This suggests a promising avenue for future research, exploring the integration or combination of multiple metrics. The findings suggest that subtle differences in workload levels may be distinguishable using combinations of subjective and physiological metrics.

Ultra‐Stretchable and Environmentally Resilient Hydrogels Via Sugaring‐Out Strategy for Soft Robotics Sensing

AbstractThe adoption of hydrogels in most applications is hampered by their high free water content, which limits their mechanical performance and environmental resilience. Herein, this issue is simultaneously addressed by modulating the state of water and the intermolecular interactions in polyacrylamide (PAM) hydrogels. Specifically, PAM hydrogels are toughened by sugaring‐out using a monosaccharide (glucose, G). Glucose is found to facilitate PAM hydrogen bonding and interchain interactions. Meanwhile, the high hygroscopicity of glucose converts some of the free water to bound state, endowing the hydrogels with remarkable resilience to extreme environmental conditions. The PAM‐G hydrogels are demonstrated as multimodal sensors for soft robotics. Moreover, PAM‐G alcogels produced by solvent exchanging with ethanol are shown as effective opto‐mechanical sensors. Notably, all these properties are obtained by the inclusion of glucose, a green additive showing no negative health and environmental effect.

Flexible Multimodal Sensing System Based on a Vertical Stacking Strategy for Efficiently Decoupling Multiple Signals.

Multisensory integration enables the simultaneous perception of multiple environmental stimuli while minimizing size and energy consumption. However, conventional multifunctional integration in flexible electronics typically requires large-scale horizontal sensing arrays (such as flexible printed circuit boards), posing decoupling complexities, tensile strain limitation, and spatial constraints. Herein, a fully flexible multimodal sensing system (FMSS) is developed by coupling biomimetic stretchable conductive films (BSCFs) and strain-insensitive communication interfaces using a vertical stacking integration strategy. The FMSS achieves vertical integration without additional adhesives, and it can incorporate individual sensing layers and stretchable interconnects without any essential constraint on their deformations. Accordingly, the temperature and pressure are precisely decoupled simultaneously, and tensile stress can be accurately discerned in different directions. This vertical stacking integration strategy is expected to offer a new approach to significantly streamline the design and fabrication of multimodal sensing systems and enhance their decoupling capabilities.

Developing a Smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors Integration and Internet of Things

Developing a Smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors Integration and Internet of Things

Multimodal Sensors Enabled Autonomous Soft Robotic System with Self-Adaptive Manipulation.

Human hands are amazingly skilled at recognizing and handling objects of different sizes and shapes. To date, soft robots rarely demonstrate autonomy equivalent to that of humans for fine perception and dexterous operation. Here, an intelligent soft robotic system with autonomous operation and multimodal perception ability is developed by integrating capacitive sensors with triboelectric sensor. With distributed multiple sensors, our robot system can not only sense and memorize multimodal information but also enable an adaptive grasping method for robotic positioning and grasp control, during which the multimodal sensory information can be captured sensitively and fused at feature level for crossmodally recognizing objects, leading to a highly enhanced recognition capability. The proposed system, combining the performance and physical intelligence of biological systems (i.e., self-adaptive behavior and multimodal perception), will greatly advance the integration of soft actuators and robotics in many fields.

Exploring deep learning techniques for wild animal behaviour classification using animal‐borne accelerometers

Abstract Machine learning‐based behaviour classification using acceleration data is a powerful tool in bio‐logging research. Deep learning architectures such as convolutional neural networks (CNN), long short‐term memory (LSTM) and self‐attention mechanism as well as related training techniques have been extensively studied in human activity recognition. However, they have rarely been used in wild animal studies. The main challenges of acceleration‐based wild animal behaviour classification include data shortages, class imbalance problems, various types of noise in data due to differences in individual behaviour and where the loggers were attached and complexity in data due to complex animal‐specific behaviours, which may have limited the application of deep learning techniques in this area. To overcome these challenges, we explored the effectiveness of techniques for efficient model training: data augmentation, manifold mixup and pre‐training of deep learning models with unlabelled data, using datasets from two species of wild seabirds and state‐of‐the‐art deep learning model architectures. Data augmentation improved the overall model performance when one of the various techniques (none, scaling, jittering, permutation, time‐warping and rotation) was randomly applied to each data during mini‐batch training. Manifold mixup also improved model performance, but not as much as random data augmentation. Pre‐training with unlabelled data did not improve model performance. The state‐of‐the‐art deep learning models, including a model consisting of four CNN layers, an LSTM layer and a multi‐head attention layer, as well as its modified version with shortcut connection, showed better performance among other comparative models. Using only raw acceleration data as inputs, these models outperformed classic machine learning approaches that used 119 handcrafted features. Our experiments showed that deep learning techniques are promising for acceleration‐based behaviour classification of wild animals and highlighted some challenges (e.g. effective use of unlabelled data). There is scope for greater exploration of deep learning techniques in wild animal studies (e.g. advanced data augmentation, multimodal sensor data use, transfer learning and self‐supervised learning). We hope that this study will stimulate the development of deep learning techniques for wild animal behaviour classification using time‐series sensor data.

Validating EmotiBit, an open-source multi-modal sensor for capturing research-grade physiological signals from anywhere on the body

Peripheral physiological signals provide a powerful window into understanding the body and mind. Peripheral physiology devices that are currently available on the market, however, face a number of challenges. Consumer-grade devices (e.g. FitBit, Apple Watch) are easy to wear, but provide limited access to unprocessed data. Combined with black-box signal processing algorithms, this makes it difficult to interpret the data for scientific research purposes. Research-grade devices (e.g. Empatica, Shimmer, BIOPAC) provide greater access to high-quality data but remain in closed ecosystems and at price points that are out of reach for many. To bridge the gaps in available biometric solutions, our labs have created an open-source physiological sensing platform called EmotiBit (http://www.emotibit.com/), measuring EDA, multi-wavelength PPG, temperature, and 9-axis IMU. This study compares EmotiBit biometric signals to gold-standard devices by Brain Products and finds that the physiological signals exhibit a high degree of similarity, validating their use in research.

CrossPrune: Cooperative pruning for camera–LiDAR fused perception models of autonomous driving

Deep neural network pruning is effective in enabling high-performance perception models to be deployed on autonomous driving platforms with limited computation and memory resources. With their rapid development, state-of-the-art autonomous driving perception (ADP) models have advocated the use of multimodal sensors to extract diverse feature categories. However, existing pruning studies in the ADP area focus on single-modal models and neglect multimodal models. Compared with conventional pruning, multimodal pruning presents a new type of redundancy, namely modal-wise redundancy that is caused by multimodal branches extracting similar perception information. When a specific type of information is extracted by more than one modal branch, although this extraction is deemed essential from an individual single-modal standpoint, it becomes redundant when viewed from a modal-wise perspective. Therefore, modal branches must be handled cooperatively to eliminate modal-wise redundancy while concurrently preserving the original perception accuracy. Building on this, we propose CrossPrune, a modal cooperative pruning framework designed for camera–LiDAR fused perception in autonomous driving. The primary objective of CrossPrune is to effectively eliminate redundancy and achieve nondestructive pruning for multimodal ADP models. This was accomplished by approaching the problem as a multi-objective optimization task, encompassing both weight pruning and the restriction of feature distortions caused by pruning. Experiments conducted on the nuScenes and KITTI datasets demonstrated that CrossPrune attained superior pruning ratios while minimizing accuracy loss, surpassing the performance of the baselines. The key results indicated that the proposed CrossPrune achieved relative improvements of 9.6% in mAP and 11.5% in NDS under 89.8% pruning sparsity.

Using bacterial cellulose to bridge covalent and physical crosslinks in hydrogels for fabricating multimodal sensors

Network optimization is vital for the polysaccharide based hydrogels with multiple crosslinks. In this study, we developed a ‘two-step’ strategy to activate synergistic effect of chemical and physical crosslinks using a poly (vinyl alcohol) (PVA)/bacterial cellulose (BC) hydrogel as a template. The BC nanofibers, on the one hand, acted as nucleating agents, participating in the crystallization of PVA, and on the other hand, were also involved in the formation of boronic ester bond, anchored with the PVA chains via chemical bonding. Therefore, the existence of BC nanofibers, as ‘bridge’, linked the crystalline regions and amorphous parts of PVA together, associating the two characteristic crosslinks, which was conducive to load transfer. The mechanical properties of resultant hydrogels, including the tensile elongation and strength, as well as fracture toughness, were significantly improved. Moreover, the dually cross-linked hydrogels possessed ionic conductivity, which was sensitive to the tensile deformation and environmental temperature. This study clarifies a unique role of BC nanofibers in hydrogels, and proposes an effective approach to construct multiple networks in the nanocellulose reinforced PVA hydrogels.