Multisensory Integration Research Articles

Integrating vision and somatosensation does not improve the accuracy and response time when estimating area and perimeter of rectangles in primary school

BackgroundProblem-solving and learning in mathematics involves sensory perception and processing. Multisensory integration may contribute by enhancing sensory estimates. This study aims to assess if combining visual and somatosensory information improves elementary students' perimeter and area estimates. Methods87 4th graders compared rectangles with respect to area or perimeter either solely using visual observation or additionally with somatosensory information. Three experiments targeted different task aspects. Statistical analyses tested success rates and response times. ResultsContrary to expectations, adding somatosensory information did not boost success rates for area and perimeter comparison. Response time even increased with adding somatosensory information. Children's difficulty in accurately tracing figures negatively impacted the success rate of area comparisons. DiscussionResults suggest visual observation alone suffices for accurately estimating and comparing area and perimeter of rectangles in 4th graders. ImplicationsCareful deliberation on the inclusion of somatosensory information in mathematical tasks concerning perimeter and area estimations of rectangles is recommended.

Multi-timescale neural dynamics for multisensory integration.

Carrying out any everyday task, be it driving in traffic, conversing with friends or playing basketball, requires rapid selection, integration and segregation of stimuli from different sensory modalities. At present, even the most advanced artificial intelligence-based systems are unable to replicate the multisensory processes that the human brain routinely performs, but how neural circuits in the brain carry out these processes is still not well understood. In this Perspective, we discuss recent findings that shed fresh light on the oscillatory neural mechanisms thatmediate multisensory integration (MI), including power modulations, phase resetting, phase-amplitude coupling and dynamic functional connectivity. We then consider studies that also suggest multi-timescale dynamics in intrinsic ongoing neural activity and during stimulus-driven bottom-up and cognitive top-down neural network processing in the context of MI. We propose a new concept of MI that emphasizes the critical role of neural dynamics at multiple timescales within and across brain networks, enabling the simultaneous integration, segregation, hierarchical structuring and selection of information in different time windows. To highlight predictions from our multi-timescale concept of MI, real-world scenarios in which multi-timescale processes may coordinate MI in a flexible and adaptive manner are considered.

Multisensory integration of speech and gestures in a naturalistic paradigm.

Speech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand this process in response to complex, naturalistic stimuli encountered in everyday life. This study investigated behavioural and neural MSI in neurotypical adults experiencing audio-visual speech within a naturalistic, social context. Our novel paradigm incorporated a broader social situational context, complete words, and speech-supporting iconic gestures, allowing for context-based pragmatics and semantic priors. We investigated MSI in the presence of unimodal (auditory or visual) or complementary, bimodal speech signals. During audio-visual speech trials, compared to unimodal trials, participants more accurately recognised spoken words and showed a more pronounced suppression of alpha power-an indicator of heightened integration load. Importantly, on the neural level, these effects surpassed mere summation of unimodal responses, suggesting non-linear MSI mechanisms. Overall, our findings demonstrate that typically developing adults integrate audio-visual speech and gesture information to facilitate speech comprehension in noisy environments, highlighting the importance of studying MSI in ecologically valid contexts.

Synaptic ring attractor: A unified framework for attractor dynamics and multiple cues integration

Effective cue integration is essential for an animal's survival. The ring attractor network has emerged as a powerful framework for understanding how animals seamlessly integrate various cues. This network not only elucidates neural dynamics within the brain, especially in spatial encoding systems like the heading direction (HD) system, but also sheds light on cue integration within decision-making processes. Yet, many significant phenomena across different fields lack clear explanations. For instance, in physiology, the integration mechanism of Drosophila's compass neuron when confronted with conflicting self-motion cues and external sensory cues with varying gain control settings is not well elucidated. Similarly, in ethology, the decision-making system shows Bayesian integration (BI) under minimal cue conflicts, but shifts to a winner-take-all (WTA) mode as conflicts surpass a certain threshold. To address these gaps, we introduce a ring attractor network with asymmetrical neural connections and synaptic dynamics in this paper. A thorough series of simulations has been conducted to assess its ability to track external cues and integrate conflicting cues. The results from these simulations demonstrate that the proposed model replicates observed neural dynamics and offers a framework for modeling biologically plausible cue integration behaviors. Furthermore, our findings yield several testable predictions that could inform future neuroethological research, providing insights into the role of ring attractor dynamics in the animal brain.

Personalized Recommendation Systems Powered By Large Language Models: Integrating Semantic Understanding and User Preferences

This study proposes a novel personalized recommendation system leveraging Large Language Models (LLMs) to integrate semantic understanding with user preferences [1]. The system addresses critical challenges in traditional recommendation approaches by harnessing LLMs' advanced natural language processing capabilities. We introduce a framework combining a fine-tuned Roberta semantic analysis model with a multi-modal user preference extraction mechanism.The LLM component undergoes domain adaptation using Masked Language Modeling on a corpus of 112,000 user reviews from the MyAnimeList dataset, followed by task-specific fine-tuning using contrastive learning. User preferences are modeled through a weighted combination of explicit ratings, review sentiment, and implicit feedback, incorporating temporal dynamics through a time-decay function. Experimental results demonstrate significant improvements over state-of-the-art baselines, including Matrix Factorization, Neural Collaborative Filtering, BERT4Rec, and LightGCN. Our LLM-powered system achieves an 8.6%increase in NDCG@10 and a 10.5% improvement in Mean Reciprocal Rank compared to the best-performing baseline. Ablation studies reveal the synergistic effect of integrating LLM-based semantic understanding with user preference modeling. Case studies highlight the system's ability to recommend long-tail items and provide cross-genre suggestions, showcasing its capacity for nuanced content understanding. Scalability analysis indicates that while the LLM-based approach has higher initial computational costs, its performance scales comparably to other deep learning models for larger datasets. This research contributes to the field by demonstrating the effectiveness of LLMs in enhancing recommendation accuracy and diversity. Future work will explore advanced LLM architectures, multi-modal data integration, and techniques to improve computational efficiency and interpretability of recommendations.

Multimodal data fusion enhanced deep learning prediction of crack path segmentation in CFRP composites

Carbon fiber-reinforced polymer (CFRP) composites are extensively used in various engineering applications due to their superior strength-to-weight ratio and excellent mechanical properties. Predicting crack propagation paths in CFRP composites is a complex challenge due to their multiphase nature and intricate microstructural interactions. While finite element (FE) simulations possess significant capabilities for this purpose, they entail substantial computational demands and extended execution times, thereby limiting their viability in applications with high computational requirements. To address this challenge, we propose an end-to-end deep learning framework specifically for predicting crack propagation paths in two-dimensional CFRP composites. Drawing inspiration from semantic segmentation techniques, we employ EfficientNet for feature extraction, enabling the capture of hierarchical and multiscale features from both microstructure images and stress field distributions. A key aspect of our framework is the utilization of multimodal data fusion and self-attention mechanisms to effectively integrate these diverse data sources. The results demonstrate the effectiveness of our multimodal feature integration approach, producing accurate segmentations of crack path. This novel framework offers a promising approach to understanding and predicting failure mechanisms in composite materials, with significant implications for the design and maintenance of advanced composite structures.

Tool-sensed object information effectively supports vision for multisensory grasping.

Tools enable humans to extend their sensing abilities beyond the natural limits of their hands, allowing them to sense objects as if they were using their hands directly. The similarities between direct hand interactions with objects (hand-based sensing) and the ability to extend sensory information processing beyond the hand (tool-mediated sensing) entail the existence of comparable processes for integrating tool- and hand-sensed information with vision, raising the question of whether tools support vision in bimanual object manipulations. Here, we investigated participants' performance while grasping objects either held with a tool or with their hand and compared these conditions with visually guided grasping (Experiment 1). By measuring reaction time, peak velocity, and peak of grip aperture, we found that actions were initiated earlier and performed with a smaller peak grip aperture when the object was seen and held with the tool or the contralateral hand compared to when it was only seen. Thus, tool-mediated sensing effectively supports vision in multisensory grasping and, even more intriguingly, resembles hand-based sensing. We excluded that results were due to the force exerted on the tool's handle (Experiment 2). Additionally, as for hand-based sensing, we found evidence that the tool supports vision by mainly providing object positional information (Experiment 3). Thus, integrating the tool-sensed position of the object with vision is sufficient to promote a multisensory advantage in grasping. Our findings indicate that multisensory integration mechanisms significantly improve grasping actions and fine-tune contralateral hand movements even when object information is only indirectly sensed through a tool. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Convolutional Neural Network-Based Digital Diagnostic Tool for the Identification of Psychosomatic Illnesses

This paper appraises convolutional neural network (CNN) models’ capabilities in emotion detection from facial expressions, seeking to aid the diagnosis of psychosomatic illnesses, typically made in clinical setups. Using the FER-2013 dataset, two CNN models were designed to detect six emotions with 64% accuracy—although not evenly distributed; they demonstrated higher effectiveness in identifying “happy” and “surprise.” The assessment was performed through several performance metrics—accuracy, precision, recall, and F1-scores—besides further validation with an additional simulated clinical environment for practicality checks. Despite showing promising levels for future use, this investigation highlights the need for extensive validation studies in clinical settings. This research underscores AI’s potential value as an adjunct to traditional diagnostic approaches while focusing on wider scope (broader datasets) plus focus (multimodal integration) areas to be considered among recommendations in forthcoming studies. This study underscores the importance of CNN models in developing psychosomatic diagnostics and promoting future development based on ethics and patient care.

Practice of Computer Aided Design and Multimodal Integration in New Media Art

Practice of Computer Aided Design and Multimodal Integration in New Media Art

Innovation of Digital Design of Intangible Cultural Heritage Based on Knowledge Graph and Multimodal Integration

Innovation of Digital Design of Intangible Cultural Heritage Based on Knowledge Graph and Multimodal Integration

Evaluating artificial intelligence for medical imaging: a primer for clinicians.

Artificial intelligence has the potential to transform medical imaging. The effective integration of artificial intelligence into clinical practice requires a robust understanding of its capabilities and limitations. This paper begins with an overview of key clinical use cases such as detection, classification, segmentation and radiomics. It highlights foundational concepts in machine learning such as learning types and strategies, as well as the training and evaluation process. We provide a broad theoretical framework for assessing the clinical effectiveness of medical imaging artificial intelligence, including appraising internal validity and generalisability of studies, and discuss barriers to clinical translation. Finally, we highlight future directions of travel within the field including multi-modal data integration, federated learning and explainability. By having an awareness of these issues, clinicians can make informed decisions about adopting artificial intelligence for medical imaging, improving patient care and clinical outcomes.

Multisensory conflict affects body schema and reaching space

Multisensory integration plays a crucial role in building the sense of body ownership, i.e., the perceptual status of one’s body for which the body is perceived as belonging to oneself. Temporal and spatial mismatching of visual and tactile signals coming from one’s body can reduce ownership feelings towards the body and its parts, i.e., produce disownership feelings. Here, we investigated whether visuo–tactile conflict also affects the sensorimotor representation of the body in space (i.e., body schema) and the perception of the space around the body in terms of action potentiality (i.e., reaching space). In two experiments, body schema (Experiment 1) and reaching space (Experiment 2) were assessed before and after either synchronous or asynchronous visuo–tactile stimulation. Results showed that the asynchronous condition, provoking multisensory conflict, caused disownership over one’s hand and concurrently affected the body schema and the reaching space. These findings indicate that body schema and reaching space could be dynamically shaped by the multisensory regularities that build up the sense of body ownership.

Multimodal data integration for oncology in the era of deep neural networks: a review.

Cancer research encompasses data across various scales, modalities, and resolutions, from screening and diagnostic imaging to digitized histopathology slides to various types of molecular data and clinical records. The integration of these diverse data types for personalized cancer care and predictive modeling holds the promise of enhancing the accuracy and reliability of cancer screening, diagnosis, and treatment. Traditional analytical methods, which often focus on isolated or unimodal information, fall short of capturing the complex and heterogeneous nature of cancer data. The advent of deep neural networks has spurred the development of sophisticated multimodal data fusion techniques capable of extracting and synthesizing information from disparate sources. Among these, Graph Neural Networks (GNNs) and Transformers have emerged as powerful tools for multimodal learning, demonstrating significant success. This review presents the foundational principles of multimodal learning including oncology data modalities, taxonomy of multimodal learning, and fusion strategies. We delve into the recent advancements in GNNs and Transformers for the fusion of multimodal data in oncology, spotlighting key studies and their pivotal findings. We discuss the unique challenges of multimodal learning, such as data heterogeneity and integration complexities, alongside the opportunities it presents for a more nuanced and comprehensive understanding of cancer. Finally, we present some of the latest comprehensive multimodal pan-cancer data sources. By surveying the landscape of multimodal data integration in oncology, our goal is to underline the transformative potential of multimodal GNNs and Transformers. Through technological advancements and the methodological innovations presented in this review, we aim to chart a course for future research in this promising field. This review may be the first that highlights the current state of multimodal modeling applications in cancer using GNNs and transformers, presents comprehensive multimodal oncology data sources, and sets the stage for multimodal evolution, encouraging further exploration and development in personalized cancer care.

Dimensional tolerance optimization of SAR antennas with uncertainty quantification and reliability analysis based on structural-electromagnetic coupling model

The dimensional tolerances greatly impact the structural-electromagnetic performance of spaceborne synthetic aperture radar (SAR) antenna. However, few works have conducted the tolerance allocation of the extendible support structure to guarantee system reliability. To bridge the above gap, an integrated optimization methodology is proposed to design the dimensional tolerances with the structural-electromagnetic coupling model in this study. First, the absolute nodal coordinate formulation is developed to predict the structural deformation triggered by the dimensional deviations. Then, the coupling relationship between the displacement field and the electromagnetic field is elucidated for the SAR system. Subsequently, taking advantage of the surrogate model and the parallel update criterion, the interval uncertainty quantification stemming from dimensional tolerances is executed to quickly identify the worst-case scenarios of structural response. Moreover, a multi-point selection strategy is incorporated into the adaptive Bayesian probabilistic integration with active learning, which significantly improves the computational efficiency of electromagnetic reliability. Based on the above tolerance analysis, the inverse optimization design of the dimensional tolerance is realized by the particle swarm algorithm. Finally, the effectiveness and superiority of the proposed methods are validated in the case study. Our innovative framework and analytical methodology not only furnish a comprehensive structural-electromagnetic coupling model but also transition the design paradigm from experiential to robust tolerance allocation.

Multiomics unveils extracellular vesicle-driven mechanisms of endothelial communication in human carotid atherosclerosis.

Background: Carotid atherosclerosis is orchestrated by cell-cell communication that drives progression along a clinical continuum (asymptomatic to symptomatic). Extracellular vesicles (EVs) are cell-derived nanoparticles representing a new paradigm in cellular communication. Little is known about their biological cargo, cellular origin/destination, and functional roles in human atherosclerotic plaque. Methods: EVs were enriched via size exclusion chromatography from human carotid endarterectomy samples dissected into paired plaque and marginal zones (symptomatic n=16, asymptomatic n=13). EV cargos were assessed via whole transcriptome miRNA sequencing and mass spectrometry-based proteomics. EV multi-omics were integrated with bulk and single cell RNA-sequencing (scRNA-seq) datasets to predict EV cellular origin and ligand-receptor interactions, and multi-modal biological network integration of EV-cargo was completed. EV functional impact was assessed with endothelial angiogenesis assays. Results: Carotid plaques contained more EVs than adjacent marginal zones, with differential enrichment for EV-miRNAs and EV-proteins in key atherogenic pathways. EV cellular origin analysis suggested that tissue EV signatures originated from endothelial cells (EC), smooth muscle cells (SMC), and immune cells. Integrated tissue vesiculomics and scRNA-seq indicated complex EV-vascular cell communication that changed with disease progression and plaque vulnerability (i.e., symptomatic disease). Plaques from symptomatic patients, but not asymptomatic patients, were characterized by increased involvement of endothelial pathways and more complex ligand-receptor interactions, relative to their marginal zones. Plaque-EVs were predicted to mediate communication with ECs. Pathway enrichment analysis delineated an endothelial signature with roles in angiogenesis and neovascularization - well-known indices of plaque instability. This was validated functionally, wherein human carotid symptomatic plaque EVs induced sprouting angiogenesis in comparison to their matched marginal zones. Conclusion: Our findings indicate that EVs may drive dynamic changes in plaques through EV- vascular cell communication and effector functions that typify vulnerability to rupture, precipitating symptomatic disease. The discovery of endothelial-directed angiogenic processes mediated by EVs creates new therapeutic avenues for atherosclerosis.

SketchDream: Sketch-based Text-To-3D Generation and Editing

Existing text-based 3D generation methods generate attractive results but lack detailed geometry control. Sketches, known for their conciseness and expressiveness, have contributed to intuitive 3D modeling but are confined to producing texture-less mesh models within predefined categories. Integrating sketch and text simultaneously for 3D generation promises enhanced control over geometry and appearance but faces challenges from 2D-to-3D translation ambiguity and multi-modal condition integration. Moreover, further editing of 3D models in arbitrary views will give users more freedom to customize their models. However, it is difficult to achieve high generation quality, preserve unedited regions, and manage proper interactions between shape components. To solve the above issues, we propose a text-driven 3D content generation and editing method, SketchDream, which supports NeRF generation from given hand-drawn sketches and achieves free-view sketch-based local editing. To tackle the 2D-to-3D ambiguity challenge, we introduce a sketch-based multi-view image generation diffusion model, which leverages depth guidance to establish spatial correspondence. A 3D ControlNet with a 3D attention module is utilized to control multi-view images and ensure their 3D consistency. To support local editing, we further propose a coarse-to-fine editing approach: the coarse phase analyzes component interactions and provides 3D masks to label edited regions, while the fine stage generates realistic results with refined details by local enhancement. Extensive experiments validate that our method generates higher-quality results compared with a combination of 2D ControlNet and image-to-3D generation techniques and achieves detailed control compared with existing diffusion-based 3D editing approaches.

L-PBF High-Throughput Data Pipeline Approach for Multi-modal Integration

Metal-based additive manufacturing requires active monitoring solutions for assessing part quality. Multiple sensors and data streams, however, generate large heterogeneous data sets that are impractical for manual assessment and characterization. In this work, an automated pipeline is developed that enables feature extraction from high-speed camera video and multi-modal data analysis. The framework removes the need for manual assessment through the utilization of deep learning techniques and training models in a weakly supervised paradigm. We demonstrate this pipeline’s capability over 700,000 high-speed camera frames. The pipeline successfully extracts melt pool and spatter geometries and links them to corresponding pyrometry, radiography, and processparameter information. 715 individual prints are examined to reveal melt pool areas that exceeds 0.07 mm2 and pyrometry signal over a threshold (375 pyrometry units) were more likely to have defects. These automated processes enable massive throughput of characterization techniques.

Design Method of Museum Cultural and Creative Products Based on Multi sensory Experience

To explore the design method of museum cultural and creative products based on multi sensory experience, improve product level and user experience, and provide new directions for the development of museum cultural and creative design. The method first introduces the cognitive principles and sensory expression methods of sensory experience, and then focuses on the study of three common multi sensory phenomena: multi sensory integration, synesthesia, and sensory compensation. It analyzes their application in the design of cultural and creative products, summarizes the transformation process of user generated multi sensory experience, and establishes a multi sensory experience cultural and creative product design process based on this. Conclusion: Product design with multi sensory experience can bring users a more comprehensive and profound experience, help promote the inheritance and development of museum culture, and improve the serious problem of homogenization in the cultural and creative industry.

Bioinspired Artificial Visual-Respiratory Synapse as Multimodal Scene Recognition System with Oxidized-Vacancies MXene.

In the pursuit of artificial neural systems, the integration of multimodal plasticity, memory retention, and perceptual functions stands as a paramount objective in achieving neuromorphic perceptual components inspired by the human brain, to emulating the neurological excitability tuning observed in human visual and respiratory collaborations. Here, an artificial visual-respiratory synapse is presented with monolayer oxidized MXene (VRSOM) exhibiting synergistic light and atmospheric plasticity. The VRSOM enables to realize facile modulation of synaptic behaviors, encompassing postsynaptic current, sustained photoconductivity, stable facilitation/depression properties, and "learning-experience" behavior. These performances rely on the privileged photocarrier trapping characteristics and the hydroxyl-preferential selectivity inherent of oxidized vacancies. Moreover, environment recognitions and multimodal neural network image identifications are achieved through multisensory integration, underscoring the potential of the VRSOM in reproducing human-like perceptual attributes. The VRSOM platform holds significant promise for hardware output of human-like mixed-modal interactions and paves the way for perceiving multisensory neural behaviors in artificial interactive devices.

Cross-modal enhancement of defensive behavior via parabigemino-collicular projections

Effective detection and avoidance from environmental threats are crucial for animals' survival. Integration of sensory cues associated with threats across different modalities can significantly enhance animals' detection and behavioral responses. However, the neural circuit-level mechanisms underlying the modulation of defensive behavior or fear response under simultaneous multimodal sensory inputs remain poorly understood. Here, we report in mice that bimodal looming stimuli combining coherent visual and auditory signals elicit more robust defensive/fear reactions than unimodal stimuli. These include intensified escape and prolonged hiding, suggesting a heightened defensive/fear state. These various responses depend on the activity of the superior colliculus (SC), while its downstream nucleus, the parabigeminal nucleus (PBG), predominantly influences the duration of hiding behavior. PBG temporally integrates visual and auditory signals and enhances the salience of threat signals by amplifying SC sensory responses through its feedback projection to the visual layer of the SC. Our results suggest an evolutionarily conserved pathway in defense circuits for multisensory integration and cross-modality enhancement.