Multisensory Integration Research Articles

Multi-modal contrastive learning of subcellular organization using DICE.

The data deluge in biology calls for computational approaches that can integrate multiple datasets of different types to build a holistic view of biological processes or structures of interest. An emerging paradigm in this domain is the unsupervised learning of data embeddings that can be used for downstream clustering and classification tasks. While such approaches for integrating data of similar types are becoming common, there is scarcer work on consolidating different data modalities such as network and image information. Here, we introduce DICE (Data Integration through Contrastive Embedding), a contrastive learning model for multi-modal data integration. We apply this model to study the subcellular organization of proteins by integrating protein-protein interaction data and protein image data measured in HEK293 cells. We demonstrate the advantage of data integration over any single modality and show that our framework outperforms previous integration approaches. Availability: https://github.com/raminass/protein-contrastive Contact: raminass@gmail.com.

1469P Development of an efficacy prediction model for concurrent chemoradiotherapy in esophageal squamous cell carcinoma using deep learning and multimodal data integration

1469P Development of an efficacy prediction model for concurrent chemoradiotherapy in esophageal squamous cell carcinoma using deep learning and multimodal data integration

BioFusionNet: Deep Learning-Based Survival Risk Stratification in ER+ Breast Cancer Through Multifeature and Multimodal Data Fusion.

Breast cancer is a significant health concern affecting millions of women worldwide. Accurate survival risk stratification plays a crucial role in guiding personalised treatment decisions and improving patient outcomes. Here we present BioFusionNet, a deep learning framework that fuses image-derived features with genetic and clinical data to obtain a holistic profile and achieve survival risk stratification of ER+ breast cancer patients. We employ multiple self-supervised feature extractors (DINO and MoCoV3) pretrained on histopathological patches to capture detailed image features. These features are then fused by a variational autoencoder and fed to a self-attention network generating patient-level features. A co-dual-cross-attention mechanism combines the histopathological features with genetic data, enabling the model to capture the interplay between them. Additionally, clinical data is incorporated using a feed-forward network, further enhancing predictive performance and achieving comprehensive multimodal feature integration. Furthermore, we introduce a weighted Cox loss function, specifically designed to handle imbalanced survival data, which is a common challenge. Our model achieves a mean concordance index of 0.77 and a time-dependent area under the curve of 0.84, outperforming state-of-the-art methods. It predicts risk (high versus low) with prognostic significance for overall survival in univariate analysis (HR=2.99, 95% CI: 1.88-4.78, p 0.005), and maintains independent significance in multivariate analysis incorporating standard clinicopathological variables (HR=2.91, 95% CI: 1.80-4.68, p 0.005).

Modulation of audiovisual integration in the left and right sides: effects of side and spatial coherency

BackgroundUsing event-related potentials (ERPs), we aimed to investigate audiovisual integration neural mechanisms during a letter identification task in the left and right sides. Unimodal (A,V) and bimodal (AV) stimuli were presented on either side, with ERPs from unimodal (A,V) stimuli on the same side being compared to those from simultaneous bimodal stimuli (AV). Non-zero results of the AV-(A + V) difference waveforms indicated audiovisual integration on the left/right side.ResultsWhen spatially coherent AV stimuli were presented on the right side, two significant ERP components in the integrated differential wave were noted. The N134 and N262, present in the first 300 ms of the AV-(A + V) integration difference wave, indicated significant audiovisual integration effects. However, when these stimuli were presented on the left side, there were no significant integration components. This audiovisual integration difference may stem from left/right asymmetry of cerebral hemisphere language processing.ConclusionsAudiovisual letter information presented on the right side was easier to integrate, process, and represent. Additionally, only one significant integrative component peaked at 140 ms in the parietal cortex for spatially non-coherent AV stimuli and provided audiovisual multisensory integration, which could be attributed to some integrative neural processes that depend on the spatial congruity of the auditory and visual stimuli.

Metamaterial‐Based Electronic Skin with Conformality and Multisensory Integration

Metamaterial‐Based Electronic Skin with Conformality and Multisensory Integration

Harnessing Artificial Intelligence in Multimodal Omics Data Integration: Paving the Path for the Next Frontier in Precision Medicine.

The integration of multiomics data with detailed phenotypic insights from electronic health records marks a paradigm shift in biomedical research, offering unparalleled holistic views into health and disease pathways. This review delineates the current landscape of multimodal omics data integration, emphasizing its transformative potential in generating a comprehensive understanding of complex biological systems. We explore robust methodologies for data integration, ranging from concatenation-based to transformation-based and network-based strategies, designed to harness the intricate nuances of diverse data types. Our discussion extends from incorporating large-scale population biobanks to dissecting high-dimensional omics layers at the single-cell level. The review underscores the emerging role of large language models in artificial intelligence, anticipating their influence as a near-future pivot in data integration approaches. Highlighting both achievements and hurdles, we advocate for a concerted effort toward sophisticated integration models, fortifying the foundation for groundbreaking discoveries in precision medicine.

Multisensory integration and motor resonance in the primary motor cortex

Humans are endowed with a motor system that resonates to speech sounds, but whether concurrent visual information from lip movements can improve speech perception at a motor level through multisensory integration mechanisms remains unknown. Therefore, the aim of the study was to explore behavioral and neurophysiological correlates of multisensory influences on motor resonance in speech perception. Motor-evoked potentials (MEPs), by single pulse transcranial magnetic stimulation (TMS) applied over the left lip muscle (orbicularis oris) representation in the primary motor cortex, were recorded in healthy participants during the presentation of syllables in unimodal (visual or auditory) or multisensory (audio-visual) congruent or incongruent conditions. At the behavioral level, subjects showed better syllable identification in the congruent audio-visual condition as compared to the unimodal conditions, hence showing a multisensory enhancement effect. Accordingly, at the neurophysiological level, increased MEPs amplitudes were found in the congruent audio-visual condition, as compared to the unimodal ones. Incongruent audio-visual syllables resulting in illusory percepts did not increase corticospinal excitability, which in fact was comparable to that induced by the real perception of the same syllable. In conclusion, seeing and hearing congruent bilabial syllables increases the excitability of the lip representation in the primary motor cortex, hence documenting that multisensory integration can facilitate speech processing by influencing motor resonance. These findings highlight the modulation role of multisensory processing showing that it can boost speech perception and that multisensory interactions occur not only within higher-order regions, but also within primary motor areas, as shown by corticospinal excitability changes.

Phishing Webpage Detection via Multi-Modal Integration of HTML DOM Graphs and URL Features Based on Graph Convolutional and Transformer Networks

Phishing Webpage Detection via Multi-Modal Integration of HTML DOM Graphs and URL Features Based on Graph Convolutional and Transformer Networks

Enhanced Agricultural Vehicle Positioning through Ultra-Wideband-Assisted Global Navigation Satellite Systems and Bayesian Integration Techniques

This paper introduces a cooperative positioning algorithm for agricultural vehicles, which uses the relative distance of the workshop to improve the performance of the Global Navigation Satellite Systems (GNSS), to improve the positioning accuracy and stability. Firstly, the extended Kalman filter (EKF) fuses the vehicle motion state data with GNSS observation data to improve the independent GNSS positioning accuracy. Subsequently, vehicle state and observation models are formulated using Bayesian theory, incorporating GNSS/UWB data with UWB tag network ranging and with GNSS positioning data among agricultural vehicles and Inter-Vehicular Ranges (IVRs). This integration addresses the significant drift issue in GNSS elevation positioning by employing a high-dimensional decoupling algorithm, standardizing the discrete elevation data, and improving the data’s continuity and predictability. A particle filter is used to refine the vehicle’s position estimation further. Finally, experiments are carried out to verify the robustness of the proposed algorithm under different working conditions.

Enhanced taxonomic identification of fusulinid fossils through image–text integration using transformer

The accurate taxonomic identification of fusulinid fossils holds significant scientific value in palaeontology, paleoecology, and palaeogeography. However, imbalanced image samples lead to the model preferring to learn features from categories with many samples while ignoring fewer sample categories, greatly reducing the prediction accuracy of fusulinid fossil identification. Moreover, the textual description of fusulinid fossils contains rich feature information. We collected and created an order fusulinid multimodal (OFM) dataset for research. We proposed a transformer-based multimodal integration framework (TMIF) using deep learning for fusulinid fossil identification. Compared to traditional neural networks, the transformer can create global dependencies between features at different locations. TMIF incorporates image and text branches dedicated to extracting features for both modalities, and a pivotal cross-modal integration module that allows visual features to learn textual semantic features sufficiently to obtain a more comprehensive feature representation. Experimental evaluation using the OFM dataset shows that TMIF achieves a prediction accuracy of 81.7%, which is a 2.8% improvement over the only image-based method. Further comparative analyses across multiple networks affirm that the TMIF performs optimally in addressing the taxonomic identification of fusulinid fossils with imbalanced samples.

EEG workload estimation and classification: a systematic review.

Electroencephalography (EEG) has evolved into an indispensable instrument for estimating cognitive workload in various domains. ML and DL techniques have been increasingly employed to develop accurate workload estimation and classification models based on EEG data. The goal of this systematic review is to compile the body of research on EEG workload estimation and classification using ML and DL approaches. The PRISMA procedures were followed in conducting the review, searches were conducted through databases at SpringerLink, ACM Digital Library, IEEE Explore, PUBMED, and Science Direct from the beginning to the end of February 16, 2024. Studies were selected based on predefined inclusion criteria. Data were extracted to capture study design, participant demographics, EEG features, ML/DL algorithms, and reported performance metrics. Out of the 125 items that emerged, 33 scientific papers were fully evaluated. The study designs, participant demographics, and EEG workload measurement and categorization techniques used in the investigations differed. SVM, CNN, and hybrid networks are examples of ML and DL approaches that were often used. Analyzing the accuracy scores achieved by different ML/DL models. Furthermore, a relationship was noted between sample frequency and model accuracy, with higher sample frequencies generally leading to improved performance. The percentage distribution of ML/DL methods revealed that SVMs, CNNs, and RNNs were the most commonly utilized techniques, reflecting their robustness in handling EEG data. The comprehensive review emphasizes how ML may be used to identify mental workload across a variety of disciplines using EEG data. Optimizing practical applications requires multimodal data integration, standardization efforts, and real-world validation studies. These systems will also be further improved by addressing ethical issues and investigating new EEG properties, which will improve human-computer interaction and performance assessment.

Different sensory information is used for state estimation when stationary or moving.

The accurate estimation of limb state is necessary for movement planning and execution. While state estimation requires both feedforward and feedback information, we focus here on the latter. Prior literature has shown that integrating visual and proprioceptive feedback improves estimates of static limb position. However, differences in visual and proprioceptive feedback delays suggest that multisensory integration could be disadvantageous when the limb is moving. We formalized this hypothesis by modeling feedback-based state estimation using the longstanding maximum likelihood estimation model of multisensory integration, which we updated to account for sensory delays. Our model predicted that the benefit of multisensory integration was largely lost when the limb was passively moving. We tested this hypothesis in a series of experiments in human subjects that compared the degree of interference created by discrepant visual or proprioceptive feedback when estimating limb position either statically at the end of the movement or dynamically at movement midpoint. In the static case, we observed significant interference: discrepant feedback in one modality systematically biased sensory estimates based on the other modality. However, no interference was seen in the dynamic case: participants could ignore sensory feedback from one modality and accurately reproduce the motion indicated by the other modality. Together, these findings suggest that the sensory feedback used to compute a state estimate differs depending on whether the limb is stationary or moving. While the former may tend toward multimodal integration, the latter is more likely to be based on feedback from a single sensory modality.Significance statement The estimation of limb state involves feedforward and feedback information. While estimation based on feedback has been well studied when the limb is stationary, it is unknown if similar sensory processing supports limb position estimates when moving. Using a computational model and behavioral experiments, we show that feedback-based state estimation may involve multisensory integration in the static case, but it is likely based on a single modality when the limb is moving. We suggest that this difference may stem from visual and proprioceptive feedback delays.

EEG Cortical Sources of Theta, Beta and Gamma Frequencies Orchestrate Audio–Visual Interactions

Background The fascinating ability of brain to integrate information from multiple sensory inputs has intrigued many researchers. Audio–visual (AV) interaction is a form of multisensory integration which we encounter to form meaningful representations of the environment around us. There is limited literature related to the underlying neural mechanisms. Purpose Quantitative EEG (QEEG), a tool with high temporal resolution can be used to understand cortical sources of AV interactions. Methods EEG data was recorded using 128 channels from 30 healthy subjects using audio, visual and AV stimuli in ‘object detection task’. Electrical source imaging was performed using s-LORETA across seven frequency bands (lower alpha 1, lower alpha 2, upper alpha, beta, delta, gamma, theta) during AV versus unimodal conditions across 66 gyri. Results The cortical sources were activated in the theta, beta, and gamma bands in cross modal versus unimodal conditions, which we propose, reflect neural communication for AV interaction network. The cortical sources constituted areas involved with visual processing, auditory processing, established multisensory (frontotemporal cortex, parietal cortex, middle temporal gyrus, superior frontal gyrus, inferior frontal gyrus, precentral gyrus) and potential multisensory areas (paracentral, postcentral and subcallosal). Conclusion Together, these results offer an integrative view of cortical areas in frequency oscillations during AV interactions.

Multimodal tactile sensing fused with vision for dexterous robotic housekeeping

As robots are increasingly participating in our daily lives, the quests to mimic human abilities have driven the advancements of robotic multimodal senses. However, current perceptual technologies still unsatisfied robotic needs for home tasks/environments, particularly facing great challenges in multisensory integration and fusion, rapid response capability, and highly sensitive perception. Here, we report a flexible tactile sensor utilizing thin-film thermistors to implement multimodal perceptions of pressure, temperature, matter thermal property, texture, and slippage. Notably, the tactile sensor is endowed with an ultrasensitive (0.05 mm/s) and ultrafast (4 ms) slip sensing that is indispensable for dexterous and reliable grasping control to avoid crushing fragile objects or dropping slippery objects. We further propose and develop a robotic tactile-visual fusion architecture that seamlessly encompasses multimodal sensations from the bottom level to robotic decision-making at the top level. A series of intelligent grasping strategies with rapid slip feedback control and a tactile-visual fusion recognition strategy ensure dexterous robotic grasping and accurate recognition of daily objects, handling various challenging tasks, for instance grabbing a paper cup containing liquid. Furthermore, we showcase a robotic desktop-cleaning task, the robot autonomously accomplishes multi-item sorting and cleaning desktop, demonstrating its promising potential for smart housekeeping.

In-Depth Analysis of GAF-Net: Comparative Fusion Approaches in Video-Based Person Re-Identification

This study provides an in-depth analysis of GAF-Net, a novel model for video-based person re-identification (Re-ID) that matches individuals across different video sequences. GAF-Net combines appearance-based features with gait-based features derived from skeletal data, offering a new approach that diverges from traditional silhouette-based methods. We thoroughly examine each module of GAF-Net and explore various fusion methods at the both score and feature levels, extending beyond initial simple concatenation. Comprehensive evaluations on the iLIDS-VID and MARS datasets demonstrate GAF-Net’s effectiveness across scenarios. GAF-Net achieves state-of-the-art 93.2% rank-1 accuracy on iLIDS-VID’s long sequences, while MARS results (86.09% mAP, 89.78% rank-1) reveal challenges with shorter, variable sequences in complex real-world settings. We demonstrate that integrating skeleton-based gait features consistently improves Re-ID performance, particularly with long, more informative sequences. This research provides crucial insights into multi-modal feature integration in Re-ID tasks, laying a foundation for the advancement of multi-modal biometric systems for diverse computer vision applications.

Stability of cross-sensory input to primary somatosensory cortex across experience.

Merging information from across sensory modalities is key to forming robust, disambiguated percepts of the world, yet how the brain achieves this feat remains unclear. Recent observations of cross-modal influences in primary sensory cortical areas have suggested that multisensory integration may occur in the earliest stages of cortical processing, but the role of these responses is still poorly understood. We address these questions by testing several hypotheses about the possible functions served by auditory influences on the barrel field of mouse primary somatosensory cortex (S1) using in vivo 2-photon calcium imaging. We observed sound-evoked spiking activity in a small fraction of cells overall, and moreover that this sparse activity was insufficient to encode auditory stimulus identity; few cells responded preferentially to one sound or another, and a linear classifier trained to decode auditory stimuli from population activity performed barely above chance. Moreover S1 did not encode information about specific audio-tactile feature conjunctions that we tested. Our ability to decode auditory audio-tactile stimuli from neural activity remained unchanged after both passive experience and reinforcement. Collectively, these results suggest that while a primary sensory cortex is highly plastic with regard to its own modality, the influence of other modalities are remarkably stable and play a largely stimulus-non-specific role.

Multimodal data integration using deep learning predicts overall survival of patients with glioma

AbstractGliomas are highly heterogenous diseases with poor prognosis. Precise survival prediction could benefit further clinical decision‐making, clinical trial incursion, and health economics. Recent research has emphasized the prognostic value of magnetic resonance imaging, pathological specimens, and circulating biomarkers. However, the integrative potential and efficacy of these modalities require to be further validated. After incorporating 218 patients of The Cancer Genome Atlas glioma datasets of and 54 patients of the Huashan cohort with complementary prognostic information, we established a squeeze‐and‐excitation deep learning feature extractor for T1‐contrast enhanced images and histological slides and explored to screen significant circulating 5‐hydroxymethylcytosines (5hmC) profiles for glioma survival by least absolute shrinkage and selection operator‐Cox regression. Therefore, a prognostication predictive model with high efficiency was obtained through survival support vector machine multimodal integration of radiologic imaging, histopathologic imaging features, genome‐wide 5hmC in circulating cell‐free DNA and clinical variables, suggesting a valid strategy (concordance‐index = 0.897; Brier score = 0.118) for improved survival risk stratification of glioma patients.

Evaluation and analysis of visual perception using attention-enhanced computation in multimedia affective computing.

Facial expression recognition (FER) plays a crucial role in affective computing, enhancing human-computer interaction by enabling machines to understand and respond to human emotions. Despite advancements in deep learning, current FER systems often struggle with challenges such as occlusions, head pose variations, and motion blur in natural environments. These challenges highlight the need for more robust FER solutions. To address these issues, we propose the Attention-Enhanced Multi-Layer Transformer (AEMT) model, which integrates a dual-branch Convolutional Neural Network (CNN), an Attentional Selective Fusion (ASF) module, and a Multi-Layer Transformer Encoder (MTE) with transfer learning. The dual-branch CNN captures detailed texture and color information by processing RGB and Local Binary Pattern (LBP) features separately. The ASF module selectively enhances relevant features by applying global and local attention mechanisms to the extracted features. The MTE captures long-range dependencies and models the complex relationships between features, collectively improving feature representation and classification accuracy. Our model was evaluated on the RAF-DB and AffectNet datasets. Experimental results demonstrate that the AEMT model achieved an accuracy of 81.45% on RAF-DB and 71.23% on AffectNet, significantly outperforming existing state-of-the-art methods. These results indicate that our model effectively addresses the challenges of FER in natural environments, providing a more robust and accurate solution. The AEMT model significantly advances the field of FER by improving the robustness and accuracy of emotion recognition in complex real-world scenarios. This work not only enhances the capabilities of affective computing systems but also opens new avenues for future research in improving model efficiency and expanding multimodal data integration.

The dissociating effects of fear and disgust on multisensory integration in autism: evidence from evoked potentials.

Deficits in Multisensory Integration (MSI) in ASD have been reported repeatedly and have been suggested to be caused by altered long-range connectivity. Here we investigate behavioral and ERP correlates of MSI in ASD using ecologically valid videos of emotional expressions. In the present study, we set out to investigate the electrophysiological correlates of audiovisual MSI in young autistic and neurotypical adolescents. We employed dynamic stimuli of high ecological validity (500 ms clips produced by actors) that depicted fear or disgust in unimodal (visual and auditory), and bimodal (audiovisual) conditions. We report robust MSI effects at both the behavioral and electrophysiological levels and pronounced differences between autistic and neurotypical participants. Specifically, neurotypical controls showed robust behavioral MSI for both emotions as seen through a significant speed-up of bimodal response time (RT), confirmed by Miller's Race Model Inequality (RMI), with greater MSI effects for fear than disgust. Adolescents with ASD, by contrast, showed behavioral MSI only for fear. At the electrophysiological level, the bimodal condition as compared to the unimodal conditions reduced the amplitudes of the visual P100 and auditory P200 and increased the amplitude of the visual N170 regardless of group. Furthermore, a cluster-based analysis across all electrodes revealed that adolescents with ASD showed an overall delayed and spatially constrained MSI effect compared to controls. Given that the variables we measured reflect attention, our findings suggest that MSI can be modulated by the differential effects on attention that fear and disgust produce. We also argue that the MSI deficits seen in autistic individuals can be compensated for at later processing stages by (a) the attention-orienting effects of fear, at the behavioral level, and (b) at the electrophysiological level via increased attentional effort.

Integrating vision and somatosensation does not improve the accuracy and response time when estimating area and perimeter of rectangles in primary school

BackgroundProblem-solving and learning in mathematics involves sensory perception and processing. Multisensory integration may contribute by enhancing sensory estimates. This study aims to assess if combining visual and somatosensory information improves elementary students' perimeter and area estimates. Methods87 4th graders compared rectangles with respect to area or perimeter either solely using visual observation or additionally with somatosensory information. Three experiments targeted different task aspects. Statistical analyses tested success rates and response times. ResultsContrary to expectations, adding somatosensory information did not boost success rates for area and perimeter comparison. Response time even increased with adding somatosensory information. Children's difficulty in accurately tracing figures negatively impacted the success rate of area comparisons. DiscussionResults suggest visual observation alone suffices for accurately estimating and comparing area and perimeter of rectangles in 4th graders. ImplicationsCareful deliberation on the inclusion of somatosensory information in mathematical tasks concerning perimeter and area estimations of rectangles is recommended.