Spatial Cues Research Articles

SemanticFormer: Hyperspectral image classification via semantic transformer

Hyperspectral image (HSI) classification is an active research problem in computer vision and multimedia field. Contrary to traditional image data, HSIs contain rich spectral, spatial and semantic information. Thus, how to extract the discriminative features for HSIs by integrating spectral, spatial and semantic cues together is the core issue to address HSI classification task. Existing works mainly focus on exploring spectral and spatial information which usually fail to fully explore the rich semantic information in HSIs. To address this issue, in this paper, we first propose a novel semantic Transformer scheme, named SemanticFormer, which aims to learn discriminative visual representations for semantics by exploiting the interaction among different semantic tokens. Using the proposed SemanticFormer, we then propose a novel heterogeneous network that contains both spectral–spatial convolution network branch and SemanticFormer branch to extract spectral–spatial and semantic features simultaneously for HSIs. Experiments on two widely used datasets demonstrate the effectiveness of our SemanticFormer and HSI classification network method. Our codes will be available in https://github.com/SissiW/SemanticFormer.

Exploring Hierarchical Spatial Layout Cues for 3D Point Cloud based Scene Graph Prediction

3D scene graph prediction is important for intelligent agents to gather information and perceive semantics of their environments. However, constructing an effective graph is nontrivial given the complexity of natural scenes. Existing solutions for graph representation of 3D scenes still distinguish each detailed discrepancy among all the relationships as flat thinking, ignoring the mechanism used by humans to perform this task. Inspired by the role of the prefrontal cortex in hierarchical reasoning, we analyze this problem from a novel perspective: exploring hierarchical spatial layout cues in 3D space and navigating that hierarchy to make the 3D scene graph more accurate in a vertical division to horizontal propagation strategy. To this end, we first encode the contextual object features for fine-gained object category classification. Next, we build a bottom-up hierarchical graph to predict remarkably diverse support relationships in a single concept regardless of numerous irrelevant relationships. Finally, equipped with the spatially-true and semantically-meaningful support relationships, we focus on the local region layout to propagate the semantic features to predict the additional non-support relationships under the guidance of the given referred hierarchical graph nodes. Experiments on the challenging 3DSSG benchmark show that our algorithm outperforms existing state-of-the-art, and can also alleviate the impact of the long-tailed distribution of training data. Our code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HHrEtvP/HSLC-3DSG/</uri> .

Camera-Based 3D Semantic Scene Completion With Sparse Guidance Network.

Semantic scene completion (SSC) aims to predict the semantic occupancy of each voxel in the entire 3D scene from limited observations, which is an emerging and critical task for autonomous driving. Recently, many studies have turned to camera-based SSC solutions due to the richer visual cues and cost-effectiveness of cameras. However, existing methods usually rely on sophisticated and heavy 3D models to process the lifted 3D features directly, which are not discriminative enough for clear segmentation boundaries. In this paper, we adopt the dense-sparse-dense design and propose a one-stage camera-based SSC framework, termed SGN, to propagate semantics from the semantic-aware seed voxels to the whole scene based on spatial geometry cues. Firstly, to exploit depth-aware context and dynamically select sparse seed voxels, we redesign the sparse voxel proposal network to process points generated by depth prediction directly with the coarse-to-fine paradigm. Furthermore, by designing hybrid guidance (sparse semantic and geometry guidance) and effective voxel aggregation for spatial geometry cues, we enhance the feature separation between different categories and expedite the convergence of semantic propagation. Finally, we devise the multi-scale semantic propagation module for flexible receptive fields while reducing the computation resources. Extensive experimental results on the SemanticKITTI and SSCBench-KITTI-360 datasets demonstrate the superiority of our SGN over existing state-of-the-art methods. And even our lightweight version SGN-L achieves notable scores of 14.80% mIoU and 45.45% IoU on SeamnticKITTI validation with only 12.5 M parameters and 7.16 G training memory. Code is available at https://github.com/Jieqianyu/SGN.

Modeling of Multiple Spatial-Temporal Relations for Robust Visual Object Tracking.

Recently, one-stream trackers have achieved parallel feature extraction and relation modeling through the exploitation of Transformer-based architectures. This design greatly improves the performance of trackers. However, as one-stream trackers often overlook crucial tracking cues beyond the template, they prone to give unsatisfactory results against complex tracking scenarios. To tackle these challenges, we propose a multi-cue single-stream tracker, dubbed MCTrack here, which seamlessly integrates template information, historical trajectory, historical frame, and the search region for synchronized feature extraction and relation modeling. To achieve this, we employ two types of encoders to convert the template, historical frames, search region, and historical trajectory into tokens, which are then collectively fed into a Transformer architecture. To distill temporal and spatial cues, we introduce a novel adaptive update mechanism, which incorporates a thresholding component and a local multi-peak component to filter out less accurate and overly disturbed tracking cues. Empirically, MCTrack achieves leading performance on mainstream benchmark datasets, surpassing the most advanced SeqTrack by 2.0% in terms of the AO metric on GOT-10k. The code is available at https://github.com/wsumel/MCTrack.

Memory Recall for Data Visualizations in Mixed Reality, Virtual Reality, 3D and 2D.

This article explores how the ability to recall information in data visualizations depends on the presentation technology. Participants viewed 10 Isotype visualizations on a 2D screen, in 3D, in Virtual Reality (VR) and in Mixed Reality (MR). To provide a fair comparison between the three 3D conditions, we used LIDAR to capture the details of the physical rooms, and used this information to create our textured 3D models. For all environments, we measured the number of visualizations recalled and their order (2D) or spatial location (3D, VR, MR). We also measured the number of syntactic and semantic features recalled. Results of our study show increased recall and greater richness of data understanding in the MR condition. Not only did participants recall more visualizations and ordinal/spatial positions in MR, but they also remembered more details about graph axes and data mappings, and more information about the shape of the data. We discuss how differences in the spatial and kinesthetic cues provided in these different environments could contribute to these results, and reasons why we did not observe comparable performance in the 3D and VR conditions.

Deep feature analysis, classification with AI-driven gastrointestinal diagnostics

Several AI-based methods have substantially progressed the area of medical image and video-based diagnostics, which encompasses radiography, pathology, endoscopy, and the categorization of gastrointestinal (GI) diseases. When it comes to classifying numerous GI disorders, the majority of prior research that relies solely on spatial cues performs poorly. While some prior research has made use of temporal features trained on a 3D convolution neural network, these studies have focused on a very small subset of the gastrointestinal system and have used very few classes. To address these concerns, we introduce an all-inclusive AI-based system for classifying different GI illnesses using endoscopic recordings. This system can extract spatial and temporal data concurrently, leading to improved classification performance. For temporal variables, we employ a long short-term memory model; for spatial features, we employ two independent residual networks in cascade mode.

Spatial release from masking in the median plane with non-native speakers using individual and mannequin head related transfer functions.

Spatial release from masking (SRM) in speech-on-speech tasks has been widely studied in the horizontal plane, where interaural cues play a fundamental role. Several studies have also observed SRM for sources located in the median plane, where (monaural) spectral cues are more important. However, a relatively unexplored research question concerns the impact of head-related transfer function (HRTF) personalisation on SRM, for example, whether using individually-measured HRTFs results in better performance if compared with the use of mannequin HRTFs. This study compares SRM in the median plane in a speech-on-speech virtual task rendered using both individual and mannequin HRTFs. SRM is obtained using English sentences with non-native English speakers. Our participants show lower SRM performances compared to those found by others using native English participants. Furthermore, SRM is significantly larger when the source is spatialised using the individual HRTF, and this effect is more marked for those with lower English proficiency. Further analyses using a spectral distortion metric and the estimation of the better-ear effect, show that the observed SRM can only partially be explained by HRTF-specific factors and that the effect of the familiarity with individual spatial cues is likely to be the most significant element driving these results.

Auditory Spatial Bisection of Blind and Normally Sighted Individuals in Free Field and Virtual Acoustics.

Sound localization is an important ability in everyday life. This study investigates the influence of vision and presentation mode on auditory spatial bisection performance. Subjects were asked to identify the smaller perceived distance between three consecutive stimuli that were either presented via loudspeakers (free field) or via headphones after convolution with generic head-related impulse responses (binaural reproduction). Thirteen azimuthal sound incidence angles on a circular arc segment of ±24° at a radius of 3 m were included in three regions of space (front, rear, and laterally left). Twenty normally sighted (measured both sighted and blindfolded) and eight blind persons participated. Results showed no significant differences with respect to visual condition, but strong effects of sound direction and presentation mode. Psychometric functions were steepest in frontal space and indicated median spatial bisection thresholds of 11°-14°. Thresholds increased significantly in rear (11°-17°) and laterally left (20°-28°) space in free field. Individual pinna and torso cues, as available only in free field presentation, improved the performance of all participants compared to binaural reproduction. Especially in rear space, auditory spatial bisection thresholds were three to four times higher (i.e., poorer) using binaural reproduction than in free field. The results underline the importance of individual auditory spatial cues for spatial bisection, irrespective of access to vision, which indicates that vision may not be strictly necessary to calibrate allocentric spatial hearing.

Externalization of Speech When Listening With Hearing Aids.

Subjective reports indicate that hearing aids can disrupt sound externalization and/or reduce the perceived distance of sounds. Here we conducted an experiment to explore this phenomenon and to quantify how frequently it occurs for different hearing-aid styles. Of particular interest were the effects of microphone position (behind the ear vs. in the ear) and dome type (closed vs. open). Participants were young adults with normal hearing or with bilateral hearing loss, who were fitted with hearing aids that allowed variations in the microphone position and the dome type. They were seated in a large sound-treated booth and presented with monosyllabic words from loudspeakers at a distance of 1.5 m. Their task was to rate the perceived externalization of each word using a rating scale that ranged from 10 (at the loudspeaker in front) to 0 (in the head) to -10 (behind the listener). On average, compared to unaided listening, hearing aids tended to reduce perceived distance and lead to more in-the-head responses. This was especially true for closed domes in combination with behind-the-ear microphones. The behavioral data along with acoustical recordings made in the ear canals of a manikin suggest that increased low-frequency ear-canal levels (with closed domes) and ambiguous spatial cues (with behind-the-ear microphones) may both contribute to breakdowns of externalization.

Coming to Refuse and Going to Stay: Attending to the Spatial Cue Effects of Repulsion and Attraction

Coming to Refuse and Going to Stay: Attending to the Spatial Cue Effects of Repulsion and Attraction

Cognitive Mechanisms Underlying Sports Performance of Athletes Engaged in Strategic Team Sports

We examined the cognitive performance of athletes engaged in strategic team sports such as football and basketball with sixty participants including athletes with at least five years of experience at National/State level in football and basketball and non-athletes. Measures of vigilance (continuous performance task), selective attention (spatial cuing task), visual search (feature and conjunction search task), attention network task, working memory (n-back task) and motion perception (apparent motion task) were employed. Athletes outperformed non- athletes on measures of vigilance, visual search, selective attention, and motion perception as they need to process multiple stimuli and engage in coordinated actions. Compared to football players, basketball players performed better with respect to vigilance, attentional facilitation and shifting; and faster target detection in conjunction search task with higher set-size. This is explained in terms of the need to assess the diverse environment and dynamically respond to the requirements related to movement sequences. Basketball and football players were comparable to non-athletes on the working memory task. Football players showed better performance on motion perception task compared to basketball players by effectively varying the temporal distance between stimuli for a strong motion percept, which may account for their ability to judge the speed of motion helping them to accurately perceive other’s actions to prepare for their own action. Thus, different strategic team sports may involve different cognitive skills mediating sports performance. Future work needs to explore more about sport- specific cognitive-perceptual expertise in athletes engaged in team sports as a function of training and experience using a longitudinal design.

Impaired orienting function detected through eye movements in patients with temporal lobe epilepsy.

Patients with temporal lobe epilepsy (TLE) often exhibit attention function impairment. The orienting network is the subsystem of the attention network that has not been fully studied. In this study, we used eye-tracking technology with an attention network test (ANT)-based task to assess the orienting function of TLE patients, aiming to characterize their eye movement patterns. A total of 37 TLE patients and 29 healthy controls (HCs) completed the ANT task based on eye-tracking technology. Orienting function damage was mainly assessed by the ANT orienting effect. Eye movement metrics, such as mean first goal-directed saccade latency (MGSL), total saccades, and saccade amplitudes, were compared between groups. The TLE patients had a significantly lower ANT orienting effect (HC, 54.05 ± 34.05; TLE, 32.29 ± 39.54) and lower eye-tracking orienting effect (HC, 116.98 ± 56.59; TLE, 86.72 ± 59.10) than those of the HCs. The larger orienting effects indicate that orienting responses are faster when receiving a spatial cue compared with a center cue. In the spatial cue condition, compared with HCs, the TLE group showed a longer first goal-directed saccade latency (HC, 76.77 ± 58.87 ms; TLE, 115.14 ± 59.15 ms), more total saccades (HC, 28.46 ± 12.30; TLE, 36.69 ± 15.13), and larger saccade amplitudes (HC, 0.75° ± 0.60°; TLE, 1.36° ± 0.89°). Furthermore, there was a positive correlation of the orienting-effect score between the ANT task and eye-tracking metrics (r = 0.58, p < 0.05). We innovatively developed a new detection method using eye-tracking technology in combination with an ANT-based task to detect the orienting function in TLE patients. The current research demonstrated that TLE patients had a significant orienting dysfunction with a specific saccade pattern characterized by a longer first goal-directed saccade latency, more total saccades, and larger saccade amplitudes. These oculomotor metrics are likely to be a better indicator of orienting function and may potentially be used for behavioral-based interventions and long-term cognition monitoring in TLE patients.

Attention shifts in the spatial cueing paradigm reflect direct influences of experience and not top-down goals.

The spatial cueing effect (SCE) that is elicited by informative spatial cues serves as an empirical marker of attention shifts in the spatial cueing paradigm, and it has been widely interpreted to reflect a relatively pure form of top-down attention control. Contrary to this interpretation, the present study examined the extent to which the magnitude of the SCE could be due to learned associations between specific cues and shifts of attention to the corresponding location, while attempting to track potential changes in participants' task goals across the experiment. This was accomplished by using a novel two-choice, spatial cueing paradigm in which participants chose between two spatial validity contexts. One spatial validity context always involved a 25%-valid (uninformative) arrow cue - called the "test" context; whereas the alternate context - called the "inducing" context - was varied between groups. In particular, associations between specific cues and shifts of attention to the corresponding cued location were perfectly predictable in the "strong inducing" context (100%-valid arrow cues) and imperfectly predictable in the "weak inducing" context (70%-valid arrow cues). Consistent with the experience-dependent account, the results showed that the magnitude of the SCE observed in the test context increased as an individual's experience with the strong inducing context increased. Furthermore, these context effects were observed using both overlearned (arrow) and arbitrary (number) cues, as well as when eye movements were controlled. Altogether, these findings suggest that the magnitude of the SCE can be influenced directly by experience, and not by the top-down goals of the individual.

Measuring 3D Video Quality of Experience (QoE) Using A Hybrid Metric Based on Spatial Resolution and Depth Cues.

A three-dimensional (3D) video is a special video representation with an artificial stereoscopic vision effect that increases the depth perception of the viewers. The quality of a 3D video is generally measured based on the similarity to stereoscopic vision obtained with the human vision system (HVS). The reason for the usage of these high-cost and time-consuming subjective tests is due to the lack of an objective video Quality of Experience (QoE) evaluation method that models the HVS. In this paper, we propose a hybrid 3D-video QoE evaluation method based on spatial resolution associated with depth cues (i.e., motion information, blurriness, retinal-image size, and convergence). The proposed method successfully models the HVS by considering the 3D video parameters that directly affect depth perception, which is the most important element of stereoscopic vision. Experimental results show that the measurement of the 3D-video QoE by the proposed hybrid method outperforms the widely used existing methods. It is also found that the proposed method has a high correlation with the HVS. Consequently, the results suggest that the proposed hybrid method can be conveniently utilized for the 3D-video QoE evaluation, especially in real-time applications.

Dynamics of space coding by mouse hippocampal CA1 field neurons in a free navigation task in different environments

The development of persistent cognitive adaptations in neurons is a key area of focus in contemporary neuroscience. A significant illustration of such adaptations occurs through spatial specializations in place cells with one or more receptive fields that are sensitive to spatial cues (place fields) [1]. In prior research, we extensively examined the dynamics of place field formation in granular layer neurons of the CA1 field in the hippocampus of mice during arbitrary free navigation in a circular track [2]. As the primary factor for maintaining stability in the spatial specialization of place cells, we have introduced dynamic selectivity. This approach enables us to monitor the latency of each place field’s formation and the selectivity dynamics of place cells to their respective fields throughout one or multiple shooting sessions. However, in one-dimensional settings, like the circular track referenced previously, place fields can demonstrate direction-specificity with regards to animal movement [3]. This factor considerably complicates the analysis for arbitrary movement trajectories of animals. In this study, additional experiments were conducted to capture the neural activity of mice in two-dimensional environments. The experiments were conducted in both a rectangular field with objects and a circular arena with a varying number of obstacles. In this paper, we conduct a comparative analysis to examine the critical parameters governing the development of spatial specializations in a one-dimensional circle track and two-dimensional arenas. These parameters include specialization latency, average dynamic selectivity, initial increase in selectivity, and the proportion of “immediate” place fields that remain stable from the first animal visit. The study discovered that the mean selectivity of the place fields escalated in each session, with higher rates observed in subsequent sessions versus the initial session in a novel setting. Additionally, there was a significant occurrence of “immediate” place fields, representing 11% of all place fields in the rounded arena with obstacles, and 25% of all place fields in the circular track. Additionally, a population analysis of neural activity was performed on the first session in a circular track and a round arena with obstacles. Using Laplace eigenmaps for the dimension reduction of population vectors, the trajectory of animals was reconstructed, and the accuracy of this reconstruction agreed with the average dynamic selectivity for each animal included in the analysis. Thus, dynamic selectivity was verified as a measure of the spatial coding quality of the entire registered population of neurons.

Effects of presentation level on speech-on-speech masking by voice-gender difference and spatial separation between talkers.

Many previous studies have reported that speech segregation performance in multi-talker environments can be enhanced by two major acoustic cues: (1) voice-characteristic differences between talkers; (2) spatial separation between talkers. Here, the improvement they can provide for speech segregation is referred to as "release from masking." The goal of this study was to investigate how masking release performance with two cues is affected by various target presentation levels. Sixteen normal-hearing listeners participated in the speech recognition in noise experiment. Speech-on-speech masking performance was measured as the threshold target-to-masker ratio needed to understand a target talker in the presence of either same- or different-gender masker talkers to manipulate the voice-gender difference cue. These target-masker gender combinations were tested with five spatial configurations (maskers co-located or 15°, 30°, 45°, and 60° symmetrically spatially separated from the target) to manipulate the spatial separation cue. In addition, those conditions were repeated at three target presentation levels (30, 40, and 50 dB sensation levels). Results revealed that the amount of masking release by either voice-gender difference or spatial separation cues was significantly affected by the target level, especially at the small target-masker spatial separation (±15°). Further, the results showed that the intersection points between two masking release types (equal perceptual weighting) could be varied by the target levels. These findings suggest that the perceptual weighting of masking release from two cues is non-linearly related to the target levels. The target presentation level could be one major factor associated with masking release performance in normal-hearing listeners.

Failure Forensics of Shaped PDC Cutters Using Image Analysis and Deep Learning

Summary One of the major advances in polycrystalline diamond compact (PDC) bits in the last 10 years is the global adoption of 3D-shaped PDC cutters. By manipulating the cutter shape based on the understandings of cutter–rock interaction mechanisms, the cutting efficiency and mechanical properties of PDC cutters have been greatly improved. Ongoing innovations in 3D-shaped PDC cutter technology are critical to overcoming the more and more challenging formations in ultradeep wells, such as the 10 000-m-deep wells being drilled in China. Such an important role for 3D-shaped PDC cutters in oil and gas drilling applications necessitates a complete and effective failure analysis method. However, the current International Association of Drilling Contractors (IADC) dull grading cannot fulfill this objective. It is out of date in judging the damages to PDC bits and exhibits more limitations in addressing the unique challenges presented by complicated cutter shapes. To address this issue, an intelligent recognition model for PDC bit damage identification was developed based on the image analysis technology and the YOLOv7 algorithm. More than 10,000 dull bit images were used to train and validate this intelligent recognition model, which were collected from 363 PDC bits that suffered different degrees of damage after being used to drill 185 wells in the Sinopec Shengli Oilfield. Compared to the existing models, the developed intelligent recognition model has several notable contributions. First, the developed model is capable of recognizing the damages of various shaped PDC cutters commonly used by the global bit manufacturers, enabling a more accurate assessment of the failure behaviors of shaped cutters and their bits. The detection accuracy of the developed model exceeds 80% based on the confusion matrix. The recognition results by the developed artificial intelligence (AI) model are consistent with the actual failure modes judged by experienced drilling engineers. Second, the developed AI model provides direct qualitative identification of the failure modes and failure reasons for both cutters and PDC bits rather than the quantitative evaluation of the missing diamond layer used by IADC dull grading. Furthermore, the developed model eliminates the effect of reclaimed cutters on the AI detection results based on the implicit use of spatial cues in the YOLOv7 algorithm. The intelligent recognition model developed in this work can provide reliable and valuable guidance for the post-run evaluation, the bit selection for the next run, and the iterative optimization of bit design.

Attentional bias in psoriasis: The role of processing time and emotional valence.

The present study explored whether people with psoriasis display an attentional bias towards disease-related threat words and whether this bias occurs relatively early during the phase of stimulus disengagement, or during a later maintained attention phase dominated by controlled strategic processes. We also explored the degree to which attentional bias is dependent on the emotional valence of control words. Individuals with psoriasis and matched controls took part in 4 online experiments. Participants completed a spatial cueing paradigm using disease-related threat words and control words as cues, in order to obtain reaction time estimates of attentional bias. We did not observe evidence for attentional bias when control words were matched with threat words for emotional valence, regardless of whether processing time for the cues was limited (Experiment 1: SOA = 250 ms) or extended (Experiment 2: SOA = 1050 ms). We also did not observe evidence for attentional bias when control words of positive valence were used, but processing time was limited (Experiment 3). An attentional bias was only observed (p = .012, Cohen's d = .37) when sufficient processing time was available and positively-valanced control words were used (Experiment 4). Rather than showing large and generalized AB effects as predicted by previous accounts, our results tentatively suggest that AB in psoriasis is restricted to situations where participants have ample processing time and threat words are easily distinguishable from control words on the basis of emotional valence. The pattern of results suggests that attentional bias in psoriasis is best characterized as a relatively slow strategic process.

Multi-modal Deep Learning Approach to Improve Sentence level Sinhala Sign Language Recognition

Sign language is used across the world for communication purposes within hearing-impaired communities. Hearing people are not well versed in sign language and most hearing-impaired are not good in general text, creating a communication barrier. Research on Sign Language Recognition (SLR) systems have shown admirable solutions for this issue. In Sri Lanka, machine learning along with neural networks has been the prominent domain of research in Sinhala SLR. All previous research is mainly focused on word-level SLR using hand gestures for translation. While this works for a certain vocabulary, there are many signs interpreted through other spatial cues like lip movements and facial expressions. Therefore, translation is limited and sometimes the interpretations can be misleading. In this research, we propose a multi-modal Deep Learning approach that can effectively recognize sentence-level sign gestures using hand and lip movements and translate to Sinhala text. The model consists of modules for visual feature extraction (ResNet), contextual relationship modeling (transformer encoder with multi-head attention), alignment (CTC) and decoding (Prefix beam search). A dataset consisting 22 of sentences used for evaluations was collected under controlled conditions for a specific day-to-day scenario (a conversation between a vendor and a customer in a shop). The proposed model achieves a best Word Error Rate (WER) of 12.70 on the testing split, improving over the single-stream model which shows a best WER of 17.41, suggesting a multi-modal approach improves overall SLR.

Enhancing learning outcomes through multisensory integration: A fMRI study of audio-visual training in virtual reality

The integration of information from different sensory modalities is a fundamental process that enhances perception and performance in real and virtual environments (VR). Understanding these mechanisms, especially during learning tasks that exploit novel multisensory cue combinations provides opportunities for the development of new rehabilitative interventions.This study aimed to investigate how functional brain changes support behavioural performance improvements during an audio-visual (AV) learning task. Twenty healthy participants underwent a 30 min daily VR training for four weeks. The task was an AV adaptation of a ‘scanning training’ paradigm that is commonly used in hemianopia rehabilitation. Functional magnetic resonance imaging (fMRI) and performance data were collected at baseline, after two and four weeks of training, and four weeks post-training.We show that behavioural performance, operationalised as mean reaction time reduction in VR, significantly improves. In separate tests in a controlled laboratory environment, we showed that the behavioural performance gains in the VR training environment transferred to a significant mean RT reduction for the trained AV voluntary task on a computer screen. Enhancements were observed in both the visual-only and AV conditions, with the latter demonstrating a faster response time supported by the presence of audio cues. The behavioural learning effect also transfers to two additional tasks that were tested: a visual search task and an involuntary visual task.Our fMRI results reveal an increase in functional activation (BOLD signal) in multisensory brain regions involved in early-stage AV processing: the thalamus, the caudal inferior parietal lobe and cerebellum. These functional changes were only observed for the trained, multisensory, task and not for unimodal visual stimulation. Functional activation changes in the thalamus were significantly correlated to behavioural performance improvements.This study demonstrates that incorporating spatial auditory cues to voluntary visual training in VR leads to augmented brain activation changes in multisensory integration, resulting in measurable performance gains across tasks. The findings highlight the potential of VR-based multisensory training as an effective method for enhancing cognitive function and as a potentially valuable tool in rehabilitative programmes.