Multimodal Detection Research Articles

Multimodal dual perception fusion framework for multimodal affective analysis

The misuse of social platforms and the difficulty in regulating post contents have culminated in a surge of negative sentiments, sarcasms, and the rampant spread of fake news. In response, Multimodal sentiment analysis, sarcasm detection and fake news detection based on image and text have attracted considerable attention recently. Due to that these areas share semantic and sentiment features and confront related fusion challenges in deciphering complex human expressions across different modalities, integrating these multimodal classification tasks that share commonalities across different scenarios into a unified framework is expected to simplify research in sentiment analysis, and enhance the effectiveness of classification tasks involving both semantic and sentiment modeling. Therefore, we consider integral components of a broader spectrum of research known as multimodal affective analysis towards semantics and sentiment, and propose a novel multimodal dual perception fusion framework (MDPF). Specifically, MDPF contains three core procedures: (1) Generating bootstrapping language-image Knowledge to enrich origin modality space, and utilizing cross-modal contrastive learning for aligning text and image modalities to understand underlying semantics and interactions. (2) Designing dynamic connective mechanism to adaptively match image-text pairs and jointly employing gaussian-weighted distribution to intensify semantic sequences. (3) Constructing a cross-modal graph to preserve the structured information of both image and text data and share information between modalities, while introducing sentiment knowledge to refine the edge weights of the graph to capture cross-modal sentiment interaction. We evaluate MDPF on three publicly available datasets across three tasks, and the empirical results demonstrate the superiority of our proposed model.

A Novel Detection and Classification Framework for Diagnosing of Cerebral Microbleeds Using Transformer and Language.

The detection of Cerebral Microbleeds (CMBs) is crucial for diagnosing cerebral small vessel disease. However, due to the small size and subtle appearance of CMBs in susceptibility-weighted imaging (SWI), manual detection is both time-consuming and labor-intensive. Meanwhile, the presence of similar-looking features in SWI images demands significant expertise from clinicians, further complicating this process. Recently, there has been a significant advancement in automated detection of CMBs using a Convolutional Neural Network (CNN) structure, aiming at enhancing diagnostic efficiency for neurologists. However, existing methods still show discrepancies when compared to the actual clinical diagnostic process. To bridge this gap, we introduce a novel multimodal detection and classification framework for CMBs' diagnosis, termed MM-UniCMBs. This framework includes a light-weight detection model and a multi-modal classification network. Specifically, we proposed a new CMBs detection network, CMBs-YOLO, designed to capture the salient features of CMBs in SWI images. Additionally, we design an innovative language-vision classification network, CMBsFormer (CF), which integrates patient textual descriptions-such as gender, age, and medical history-with image data. The MM-UniCMBs framework is designed to closely align with the diagnostic workflow of clinicians, offering greater interpretability and flexibility compared to existing methods. Extensive experimental results show that MM-UniCMBs achieves a sensitivity of 94% in CMBs' classification and can process a patient's data within 5 s.

Histopathology in focus: a review on explainable multi-modal approaches for breast cancer diagnosis.

Precision and timeliness in breast cancer detection are paramount for improving patient outcomes. Traditional diagnostic methods have predominantly relied on unimodal approaches, but recent advancements in medical data analytics have enabled the integration of diverse data sources beyond conventional imaging techniques. This review critically examines the transformative potential of integrating histopathology images with genomic data, clinical records, and patient histories to enhance diagnostic accuracy and comprehensiveness in multi-modal diagnostic techniques. It explores early, intermediate, and late fusion methods, as well as advanced deep multimodal fusion techniques, including encoder-decoder architectures, attention-based mechanisms, and graph neural networks. An overview of recent advancements in multimodal tasks such as Visual Question Answering (VQA), report generation, semantic segmentation, and cross-modal retrieval is provided, highlighting the utilization of generative AI and visual language models. Additionally, the review delves into the role of Explainable Artificial Intelligence (XAI) in elucidating the decision-making processes of sophisticated diagnostic algorithms, emphasizing the critical need for transparency and interpretability. By showcasing the importance of explainability, we demonstrate how XAI methods, including Grad-CAM, SHAP, LIME, trainable attention, and image captioning, enhance diagnostic precision, strengthen clinician confidence, and foster patient engagement. The review also discusses the latest XAI developments, such as X-VARs, LeGrad, LangXAI, LVLM-Interpret, and ex-ILP, to demonstrate their potential utility in multimodal breast cancer detection, while identifying key research gaps and proposing future directions for advancing the field.

Iterative Optimization-Enhanced Contrastive Learning for Multimodal Change Detection

Multimodal change detection (MCD) harnesses multi-source remote sensing data to identify surface changes, thereby presenting prospects for applications within disaster management and environmental surveillance. Nonetheless, disparities in imaging mechanisms across various modalities impede the direct comparison of multimodal images. In response, numerous methodologies employing deep learning features have emerged to derive comparable features from such images. Nevertheless, several of these approaches depend on manually labeled samples, which are resource-intensive, and their accuracy in distinguishing changed and unchanged regions is not satisfactory. In addressing these challenges, a new MCD method based on iterative optimization-enhanced contrastive learning is proposed in this paper. With the participation of positive and negative samples in contrastive learning, the deep feature extraction network focuses on extracting the initial deep features of multimodal images. The common projection layer unifies the deep features of two images into the same feature space. Then, the iterative optimization module expands the differences between changed and unchanged areas, enhancing the quality of the deep features. The final change map is derived from the similarity measurements of these optimized features. Experiments conducted across four real-world multimodal datasets, benchmarked against eight well-established methodologies, incontrovertibly illustrate the superiority of our proposed approach.

A Novel Framework for Multimodal Brain Tumor Detection with Scarce Labels.

Brain tumor detection has advanced significantly with the development of deep learning technology. Although multimodal data, such as Magnetic Resonance Imaging (MRI) and Computed Tomography (CT), has potential advantages in diagnostics, most existing studies rely solely on a single modality. This is because common fusion methods may lead to the loss of critical information when attempting multimodal fusion. Therefore, effectively integrating multimodal data has become a significant challenge. Additionally, medical image analysis requires large amounts of annotated data, and labeling images is a resourceintensive task that demands experienced professionals to spend a considerable amount of time. To address these challenges, this paper introduces a new unsupervised learning framework named Double-SimCLR. This framework builds on the foundation of contrastive learning and features a dual-branch structure, enabling direct and simultaneous processing of MRI and CT images for multimodal feature fusion. Given the "weak feature" characteristics of CT images (e.g., low soft tissue contrast and low resolution), we incorporated adaptive weight masking technology to enhance CT feature extraction. Moreover, we introduced a multimodal attention mechanism, which ensures that the model focuses on salient information, thereby elevating the precision and robustness of brain tumor detection. Even without substantial labeled data, experimental results demonstrate that Double-SimCLR achieves 93.458% accuracy, 92.463% precision, and a 93.058% F1-score, outperforming state-of-the-art (SOTA) models by 2.871%, 2.643%, and 3.098%, respectively.

"Hedgehog Ball"-Shaped Nanoprobes for Multimodal Detection and Imaging of Inflammatory Markers in Osteosarcoma Using Fluorescence and Electrochemiluminescence.

Inflammation can affect the progression of cancer at tumor sites, such as in osteosarcoma, by intensifying metastasis and complicating outcomes. The current diagnostic methods lack the specificity and sensitivity required for early and accurate detection, particularly in differentiating between inflammation-induced changes and tumor activities. To address this, a novel "hedgehog ball"-shaped nanoprobe, Fe3O4@Au-pep-CQDs, was developed and designed to enhance the detection of caspase-1, a key marker of inflammation. This magnetic nanoprobe facilitates simultaneous fluorescence (FL) and electrochemiluminescence (ECL) detection. Magnetic separation minimizes the quenching of nanoparticles in solution and eliminates the need for frequent electrode replacement in ECL tests, thereby simplifying diagnostic procedures. The experimental results showed that in the detection of caspase-1, the nanoprobe had a detection limit of 0.029 U/mL (FL) and 0.033 U/mL (ECL) and had a dynamic range of 0.05 to 1.0 U/mL. Additionally, the nanoprobe achieved high recovery rates of 94.36 to 102.44% (FL) and 94.36-100.12% (ECL) in spiked biological samples. Furthermore, the nanoprobe's capabilities were extended to in vivo bioimaging to provide direct, intuitive visualization of biological processes. These novel nanoprobes were able to significantly enhance the accurate detection of inflammation at tumor sites, thereby optimizing both diagnostic and therapeutic strategies.

ACDF-YOLO: Attentive and Cross-Differential Fusion Network for Multimodal Remote Sensing Object Detection

Object detection in remote sensing images has received significant attention for a wide range of applications. However, traditional unimodal remote sensing images, whether based on visible light or infrared, have limitations that cannot be ignored. Visible light images are susceptible to ambient lighting conditions, and their detection accuracy can be greatly reduced. Infrared images often lack rich texture information, resulting in a high false-detection rate during target identification and classification. To address these challenges, we propose a novel multimodal fusion network detection model, named ACDF-YOLO, basedon the lightweight and efficient YOLOv5 structure, which aims to amalgamate synergistic data from both visible and infrared imagery, thereby enhancing the efficiency of target identification in remote sensing imagery. Firstly, a novel efficient shuffle attention module is designed to assist in extracting the features of various modalities. Secondly, deeper multimodal information fusion is achieved by introducing a new cross-modal difference module to fuse the features that have been acquired. Finally, we combine the two modules mentioned above in an effective manner to achieve ACDF. The ACDF not only enhances the characterization ability for the fused features but also further refines the capture and reinforcement of important channel features. Experimental validation was performed using several publicly available multimodal real-world and remote sensing datasets. Compared with other advanced unimodal and multimodal methods, ACDF-YOLO separately achieved a 95.87% and 78.10% mAP0.5 on the LLVIP and VEDAI datasets, demonstrating that the deep fusion of different modal information can effectively improve the accuracy of object detection.

Multi-modality artificial intelligence-based transthyretin amyloid cardiomyopathy detection in patients with severe aortic stenosis.

Transthyretin amyloid cardiomyopathy (ATTR-CM) is a frequent concomitant condition in patients with severe aortic stenosis (AS), yet it often remains undetected. This study aims to comprehensively evaluate artificial intelligence-based models developed based on preprocedural and routinely collected data to detect ATTR-CM in patients with severe AS planned for transcatheter aortic valve implantation (TAVI). In this prospective, single-center study, consecutive patients with AS were screened with [99mTc]-3,3-diphosphono-1,2-propanodicarboxylic acid ([99mTc]-DPD) for the presence of ATTR-CM. Clinical, laboratory, electrocardiogram, echocardiography, invasive measurements, 4-dimensional cardiac CT (4D-CCT) strain data, and CT-radiomic features were used for machine learning modeling of ATTR-CM detection and for outcome prediction. Feature selection and classifier algorithms were applied in single- and multi-modality classification scenarios. We split the dataset into training (70%) and testing (30%) samples. Performance was assessed using various metrics across 100 random seeds. Out of 263 patients with severe AS (57% males, age 83 ± 4.6years) enrolled, ATTR-CM was confirmed in 27 (10.3%). The lowest performances for detection of concomitant ATTR-CM were observed in invasive measurements and ECG data with area under the curve (AUC) < 0.68. Individual clinical, laboratory, interventional imaging, and CT-radiomics-based features showed moderate performances (AUC 0.70-0.76, sensitivity 0.79-0.82, specificity 0.63-0.72), echocardiography demonstrated good performance (AUC 0.79, sensitivity 0.80, specificity 0.78), and 4D-CT-strain showed the highest performance (AUC 0.85, sensitivity 0.90, specificity 0.74). The multi-modality model (AUC 0.84, sensitivity 0.87, specificity 0.76) did not outperform the model performance based on 4D-CT-strain only data (p-value > 0.05). The multi-modality model adequately discriminated low and high-risk individuals for all-cause mortality at a mean follow-up of 13 months. Artificial intelligence-based models using collected pre-TAVI evaluation data can effectively detect ATTR-CM in patients with severe AS, offering an alternative diagnostic strategy to scintigraphy and myocardial biopsy.

Multi-modal sarcasm detection using ensemble net model

Multi-modal sarcasm detection using ensemble net model

Multimodal Social Media Fake News Detection Based on 1D-CCNet Attention Mechanism

Due to the explosive rise of multimodal content in online social communities, cross-modal learning is crucial for accurate fake news detection. However, current multimodal fake news detection techniques face challenges in extracting features from multiple modalities and fusing cross-modal information, failing to fully exploit the correlations and complementarities between different modalities. To address these issues, this paper proposes a fake news detection model based on a one-dimensional CCNet (1D-CCNet) attention mechanism, named BTCM. This method first utilizes BERT and BLIP-2 encoders to extract text and image features. Then, it employs the proposed 1D-CCNet attention mechanism module to process the input text and image sequences, enhancing the important aspects of the bimodal features. Meanwhile, this paper uses the pre-trained BLIP-2 model for object detection in images, generating image descriptions and augmenting text data to enhance the dataset. This operation aims to further strengthen the correlations between different modalities. Finally, this paper proposes a heterogeneous cross-feature fusion method (HCFFM) to integrate image and text features. Comparative experiments were conducted on three public datasets: Twitter, Weibo, and Gossipcop. The results show that the proposed model achieved excellent performance.

Glioma Identification Based on Digital Multimodal Spectra Integrated With Deep Learning Feature Fusion Using a Miniature Raman Spectrometer.

The miniature fiber Raman spectroscopy detection technology can reflect the properties of biomolecules through spectral characteristics and has the advantages of noninvasiveness, real-time, safety, label-free operation, and potential for early cancer diagnosis. This technology holds promise for developing portable, low-cost, intraoperative tumor detection instruments. Glioma is one of the most common malignant tumors of the central nervous system with rapid growth and a short disease course. However, the considerable heterogeneity of the glioma sample leads to substantial intraclass variance in collected spectra, coupled with the miniature Raman spectrometer's low signal-to-noise ratio. These factors diminish the accuracy of the brain glioma recognition model. To address this issue, a glioma identification method based on digital multimodal spectra integrated with deep learning features fusion (DMS-DLFF) using the miniature Raman spectrometer is proposed. Different from existing multimodal tumor detection methods employing multiple spectral instruments, DMS-DLFF enhances tumor identification accuracy without increasing hardware costs. The method mathematically decomposes the original spectra to Raman and fluorescence spectra, so as to augment the biospectral information. Then, the deep learning method is used to extract the feature information of the two kinds of spectra, respectively, and the digital multimodal spectral fusion is realized at the feature level. Moreover, a two-layer pattern recognition model is constructed based on the ensemble strategy, amalgamating the strengths of diverse classifiers. Meanwhile, the bagging strategy is introduced to improve support vector machine algorithms, one of the basic classifiers. Compared with traditional methodologies, DMS-DLFF operates at both the feature level and decision level, employing high-information-density feature vectors to train ensemble classification models for increasing overall recognition accuracy. This study collected 260 Raman spectra of glioma and 151 Raman spectra of normal brain tissue. The accuracy, sensitivity, and specificity were 91.9%, 96.7%, and 80.8%, respectively. The proposed method outperforms traditional algorithms in brain glioma detection, which helps doctors formulate precise surgical plans and thereby improve patient prognosis.

Comparison between USPIOs and SPIOs for Multimodal Imaging of Extracellular Vesicles Extracted from Adipose Tissue-Derived Adult Stem Cells.

Adipose tissue-derived adult stem (ADAS) cells and extracellular vesicle (EV) therapy offer promising avenues for treating neurodegenerative diseases due to their accessibility and potential for autologous cell transplantation. However, the clinical application of ADAS cells or EVs is limited by the challenge of precisely identifying them in specific regions of interest. This study compares two superparamagnetic iron oxide nanoparticles, differing mainly in size, to determine their efficacy for allowing non-invasive ADAS tracking via MRI/MPI and indirect labeling of EVs. We compared a USPIO (about 5 nm) with an SPIO (Resovist®, about 70 nm). A physicochemical characterization of nanoparticles was conducted using DLS, TEM, MRI, and MPI. ADAS cells were labeled with the two nanoparticles, and their viability was assessed via MTT assay. MRI detected labeled cells, while TEM and Prussian Blue staining were employed to confirm cell uptake. The results revealed that Resovist® exhibited higher transversal relaxivity value than USPIO and, consequently, allows for detection with higher sensitivity by MRI. A 200 µgFe/mL concentration was identified as optimal for ADAS labeling. MPI detected only Resovist®. The findings suggest that Resovist® may offer enhanced detection of ADAS cells and EVs, making it suitable for multimodal imaging. Preliminary results obtained by extracting EVs from ADAS cells labeled with Resovist® indicate that EVs retain the nanoparticles, paving the way to an efficient and multimodal detection of EVs.

Multimodal optical detection, identification and reduction of Cr(VI) with carbon dots and its PVA/polyNIPAM spheres based 2D luminescent photonic structures

Multiple emissive carbon dots (CDs) were synthesized through a one-step solvothermal acid-assisted method using citric acid and urea. Three main emission bands were observed at ∼431 nm, ∼501 nm, and ∼622 nm, corresponding to excitation wavelengths of 360 nm, 430 nm, and 550 nm, respectively. Simultaneous detection through salient discernible features of three metal ions has been achieved. Emission of CDs with Fe(III) shows reduction in intensity at 360 nm and 550 nm wavelengths and an enhancement of photoluminescence (PL) intensity at an excitation wavelength of 430 nm. Al(III) revealed turn-on PL signals, along with a redshift when excited at 360 and 430 nm, and turn-off intensity at excitation of 550 nm. Under all the excitation wavelengths, Cr(VI) exhibits PL quenching accompanied with an additional red shift. This sensor enhances the accuracy and reliability of the probe through multiple distinctive and characteristic responses for all the three ions for which the probes were selective. Apart from the optical detection at a certain wavelength, the Cr(VI) reduction to the less toxic Cr(III) through as-prepared CDs could be monitored at three different excitation wavelengths. The Fe(III) and Cr(VI) detection in solid state has also been demonstrated through 2D luminescent photonic structures formed with CDs-PVA (poly-vinyl alcohol)-polyNIPAM (poly (N-isopropylacrylamide)) composites that can possibly be used for anticounterfeiting applications.

Multimodal Sarcasm Detection: A Comprehensive Review

Sarcasm, a form of verbal irony, involves using remarks that intentionally convey the opposite of their literal meaning. It serves to criticize or humorously undermine a situation. Verbal and non-verbal cues—such as changes in tone, incongruence across modalities, and word emphasis—often convey sarcasm. As technology advances, more people express their opinions online, necessitating the development of efficient models for detecting speech nature. Machine Learning algorithms, Linguistic Models, Ensemble Learning, and Multi-modal approaches play crucial roles in this endeavor. While most research has focused on unimodal approaches using textual data, the field now embraces multimodal sarcasm detection. This approach integrates diverse data types, including images, text, and audio, to enhance accuracy and automate sarcasm classification. In our review paper, we delve into the captivating world of multimodal sarcasm detection, tracing its evolution from early unimodal methods to the current state of multimodal techniques. We aim to provide a comprehensive overview, shedding light on challenges, methodologies, and prospects. Keywords: Sarcasm Detection, Multimodal Approaches, Machine Learning Algorithms, Linguistic Models, Ensemble Learning

Decentralized food safety and authentication on cellulose paper-based analytical platform: A review.

Food safety and authenticity analysis play a pivotal role in guaranteeing food quality, safeguarding public health, and upholding consumer trust. In recent years, significant social progress has presented fresh challenges in the realm of food analysis, underscoring the imperative requirement to devise innovative and expedient approaches for conducting on-site assessments. Consequently, cellulose paper-based devices (PADs) have come into the spotlight due to their characteristics of microchannels and inherent capillary action. This review summarizes the recent advances in cellulose PADs in various food products, comprising various fabrication strategies, detection methods such as mass spectrometry and multi-mode detection, sampling and processing considerations, as well as applications in screening food safety factors and assessing food authenticity developed in the past 3 years. According to the above studies, cellulose PADs face challenges such as limited sample processing, inadequate multiplexing capabilities, and the requirement for workflow integration, while emerging innovations, comprising the use of simplified sample pretreatment techniques, the integration of advanced nanomaterials, and advanced instruments such as portable mass spectrometer and the innovation of multimodal detection methods, offer potential solutions and are highlighted as promising directions. This review underscores the significant potential of cellulose PADs in facilitating decentralized, cost-effective, and simplified testing methodologies to maintain food safety standards. With the progression of interdisciplinary research, cellulose PADs are expected to become essential platforms for on-site food safety and authentication analysis, thereby significantly enhancing global food safety for consumers.

Drone-Based Visible–Thermal Object Detection with Transformers and Prompt Tuning

The use of unmanned aerial vehicles (UAVs) for visible–thermal object detection has emerged as a powerful technique to improve accuracy and resilience in challenging contexts, including dim lighting and severe weather conditions. However, most existing research relies on Convolutional Neural Network (CNN) frameworks, limiting the application of the Transformer’s attention mechanism to mere fusion modules and neglecting its potential for comprehensive global feature modeling. In response to this limitation, this study introduces an innovative dual-modal object detection framework called Visual Prompt multi-modal Detection (VIP-Det) that harnesses the Transformer architecture as the primary feature extractor and integrates vision prompts for refined feature fusion. Our approach begins with the training of a single-modal baseline model to solidify robust model representations, which is then refined through fine-tuning that incorporates additional modal data and prompts. Tests on the DroneVehicle dataset show that our algorithm achieves remarkable accuracy, outperforming comparable Transformer-based methods. These findings indicate that our proposed methodology marks a significant advancement in the realm of UAV-based object detection, holding significant promise for enhancing autonomous surveillance and monitoring capabilities in varied and challenging environments.

Nanotechnology Advances in the Detection and Treatment of Lymphoid Malignancies.

Lymphoid malignancies are complex diseases with distinct biological behaviors, clinical presentations, and treatment responses. Ongoing research and advancements in biotechnology enhance the understanding and management of these malignancies, moving towards more personalized approaches for diagnosis and treatment. Nanotechnology has emerged as a promising tool to improve some limitations of conventional diagnostics as well as treatment strategies for lymphoid malignancies. Nanoparticles (NPs) offer unique advantages such as enhanced multimodal detection, drug delivery, and targeted therapy capabilities, with the potential to improve precision medicine and patient outcomes. Here, we comprehensively examine the current landscape of nanoconstructs applied in the management of lymphoid disease. Through a comprehensive analysis of preclinical studies, we highlight the translational potential of NPs in revolutionizing the field of hematological malignancies, with a specific focus on lymphoid neoplasms.

FDIA localization and classification detection in smart grids using multi-modal data and deep learning technique

False data injection attacks (FDIA) exploit the vulnerabilities of bad data detection in energy management systems to maliciously tamper with state estimation results, seriously jeopardizing the safe and reliable operation of power systems. In order to promptly detect FDIA, recent studies have used machine learning techniques to extract attack characteristics and detect FDIA based on changes in these characteristics. Current research predominantly focuses on detecting a single type of FDIA attack model and topology. However, the increasing diversification of FDIA construction methods and the variability of topological structures in power systems can reduce or even invalidate detection effectiveness. To cope with the abovementioned difficulties, this study introduces a multimodal deep learning detection model based on variational graph auto-encoders (VGAE), temporal convolutional networks (TCN), and gated recurrent units (GRU). The topological features of power systems are obtained through VGAE to adapt to the detection needs of various topological structures and performance enhancement. These features are then integrated with preprocessed measurement data via BDD to form multimodal data. Furthermore, this multimodal data is used to locate and classify FDIA with complete information, FDIA with incomplete information, and topology attack after applying a multi-label classification algorithm based on TCN-GRU for temporal feature extraction. Experiments carried out on the IEEE 14 and IEEE 118 bus systems show that the proposed method is robust and has high detection performance. The results indicate that the proposed detection method achieves AUC values that are, on average, 0.085, 0.145, and 0.04 higher than those of CNN, LSTM, and CNN-LSTM, respectively. Moreover, under various noise and attack intensities, recall, precision, F1 score, and RACC remain above 0.859, 0.877, 0.867, and 0.818, respectively, with a classification accuracy greater than 0.912. This study provides a unique perspective on detecting FDIA across various attack models and power system topologies.

Multimodal Intelligent Flooring System for Advanced Smart-Building Monitoring and Interactions.

The floor constitutes one of the largest areas within a building with which users interact most frequently in daily activities. Employing floor sensors is vital for smart-building digital twins, wherein triboelectric nanogenerators demonstrate wide application potential due to their good performance and self-powering characteristics. However, their sensing stability, reliability, and multimodality require further enhancement to meet the rapidly evolving demands. Thus, this work introduces a multimodal intelligent flooring system, implementing a 4× 4 floor array for multimodal information detection (including position, pressure, material, user identity, and activity) and human-machine interactions. The floor unit incorporates a hybrid structure of triboelectric pressure sensors and a top-surface material sensor, facilitating linear and enhanced sensitivity across a wide pressure range (0-800 N), alongside the material recognition capability. The floor array is implemented by an advanced output-ratio method with minimalist output channels, which is insensitive to environmental factors such as humidity and temperature. In addition to multimodal sensing, energy harvesting is co-designed with the pressure sensors for scavenging waste energy to power smart-building sensor nodes. This developed flooring system enables multimodal sensing, energy harvesting, and smart-sport interactions in smart buildings, greatly expanding the floor sensing scenarios and applications.

Electromechanically stable and flexible poro-hyperelastic composites for multimodal robotic tactile sensing

The expansion of automated production places increased demands on intelligent sensory networks for collecting and processing various input signals, which necessitate higher standards for multimodal detection, sensitivity, and sensing response speed. Here, this work developed a poro-hyperelastic composites-based multimodal tactile sensor capable of triboelectric, piezoresistive, and capacitive modes across multiple strains and frequency domains. Such the multilayered flexible multimodal mechano-electronic sensor (MMES) is constructed from a carbon nanotubes porous sponge (CNPS), facilitating switching between trimodal sensing modes integrated within a single sensor structure, effectively enhancing the electromechanical robustness of the sensory system. The present MMES sensor exhibits excellent pressure sensing performances, with a wide range (0–20 kPa), fast response (less than 102 ms in piezoresistive mode and less than 62 ms in capacitive mode), high sensitivity (up to 0.12 V/kPa⁻1 in triboelectric mode), and long-term durability (exceeding 10,000 cycles). Moreover, a micro-CT generated 3D model was established using finite element analysis (FEA) to investigate porous hyperelastic behaviours of the CNPS composites, in agreement with experimental data. The results demonstrated that the MMES sensor is particularly suitable for dynamic feedback in robotic arms, allowing production lines to make accurate distinguishments and operations by sensing distinct materials and resistances encountered during assembly in real time. The outcomes offer promising applications of multimodal robotic tactile sensing for human-machine interaction.