Multisensory Representation Research Articles

Mindfulness Affects the Boundaries of Bodily Self-Representation: The Effect of Focused-Attention Meditation in Fading the Boundary of Peripersonal Space.

Peripersonal space (PPS) is a dynamic multisensory representation of the space around the body, influenced by internal and external sensory information. The malleability of PPS boundaries, as evidenced by their expansion after tool use or modulation through social interactions, positions PPS as a crucial element in understanding the subjective experiences of self and otherness. Building on the existing literature highlighting both the cognitive and bodily effects of mindfulness meditation, this study proposes a novel approach by employing focused-attention meditation (FAM) and a multisensory audio-tactile task to assess PPS in both the extension and sharpness of its boundaries. The research hypothesis posits that FAM, which emphasizes heightened attention to bodily sensations and interoception, may reduce the extension of PPS and make its boundaries less sharp. We enrolled 26 non-meditators who underwent a repeated measure design in which they completed the PPS task before and after a 15-min FAM induction. We found a significant reduction in the sharpness of PPS boundaries but no significant reduction in PPS extension. These results provide novel insights into the immediate effects of FAM on PPS, potentially shedding light on the modulation of self-other representations in both cognitive and bodily domains. Indeed, our findings could have implications for understanding the intricate relationship between mindfulness practices and the subjective experience of self within spatial contexts.

Space for power: feeling powerful over others’ behavior affects peri-personal space representation

We investigated whether and how social power affects the representation of peri-personal space (PPS). We applied a multisensory interaction task to assess PPS representation and the Personal Sense of Power Scale to assess participants’ feelings of power over others’ behaviors and over others’ opinions. In Study 1, we probed PPS representation in a virtual social context. Participants with a higher sense of power showed a less defined differentiation between the close and far space as compared to participants with a lower sense of power. This effect was replicated in Study 2 when participants performed the task in a non-social context (with no person in the scene), but only after they were reminded of an episode of power. Thus, social power—the perception of power over others’ behavior—affects the multisensory representation of the self in space by blurring the differentiation between one’s own PPS and the space of others.

Multisensory perception constrains the formation of object categories: a review of evidence from sensory-driven and predictive processes on categorical decisions.

Although object categorization is a fundamental cognitive ability, it is also a complex process going beyond the perception and organization of sensory stimulation. Here we review existing evidence about how the human brain acquires and organizes multisensory inputs into object representations that may lead to conceptual knowledge in memory. We first focus on evidence for two processes on object perception, multisensory integration of redundant information (e.g. seeing and feeling a shape) and crossmodal, statistical learning of complementary information (e.g. the 'moo' sound of a cow and its visual shape). For both processes, the importance attributed to each sensory input in constructing a multisensory representation of an object depends on the working range of the specific sensory modality, the relative reliability or distinctiveness of the encoded information and top-down predictions. Moreover, apart from sensory-driven influences on perception, the acquisition of featural information across modalities can affect semantic memory and, in turn, influence category decisions. In sum, we argue that both multisensory processes independently constrain the formation of object categories across the lifespan, possibly through early and late integration mechanisms, respectively, to allow us to efficiently achieve the everyday, but remarkable, ability of recognizing objects. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Corrigendum: Design and evaluation of a multi-sensory representation of scientific data

Corrigendum: Design and evaluation of a multi-sensory representation of scientific data

Improving memory for unusual events with wakeful reactivation.

Memory consists of multiple processes, from encoding information, consolidating it into short- and long- term memory, and later retrieving relevant information. Targeted memory reactivation is an experimental method during which sensory components of a multisensory representation (such as sounds or odors) are 'reactivated', facilitating the later retrieval of unisensory attributes. We examined whether novel and unpredicted events benefit from reactivation to a greater degree than normal stimuli. We presented participants with everyday objects, and 'tagged' these objects with sounds (e.g., animals and their matching sounds) at different screen locations. 'Oddballs' were created by presenting unusual objects and sounds (e.g., a unicorn with a heartbeat sound). During a short reactivation phase, participants listened to a replay of normal and oddball sounds. Participants were then tested on their memory for visual and spatial information in the absence of sounds. Participants were better at remembering the oddball objects compared to normal ones. Importantly, participants were also better at recalling the locations of oddball objects whose sounds were reactivated, compared to objects whose sounds that were not presented again. These results suggest that episodic memory benefits from associating objects with unusual cues, and that reactivating those cues strengthen the entire multisensory representation, resulting in enhanced memory for unisensory attributes.

Design and evaluation of a multi-sensory representation of scientific data

IntroductionModern sciences and Astrophysics in particular study objects and phenomena not visible in physical terms, that is they cannot be investigated with the eyes or analogous optical systems. Nevertheless, they make intensive use of visual representations, showing data in a figurative way, using lights, shadows, colors, and shapes familiar to the user and aesthetically pleasant. Besides being inaccessible for Blind and Visually Impaired (BVI) users, such figurative visual representation can lead to misunderstandings about the real nature of the represented object, physically invisible, particularly if the representation code is not declared. In this work we argue that multi-sensory representations clearly arbitrary, i.e., that do not aim to imitate reality, are a valid choice for an effective learning and understanding of astronomical science and for an effective meaning-making process for all. The presence of a cognitively challenging code of representation can make the fruition more conscious and attentive, leading to a deeper understanding of the represented reality, also mirroring the search for relevant information proper of basic research. In an equity perspective, multi-sensory representations also create an effective common ground for inclusion among people with diverse abilities, skills, and learning styles, in the framework of Universal Design for Learning.MethodsIn order to investigate our hypothesis we designed two mono-sensory representation (one only haptic and the other only acoustic) and tested them in individual and group workshops with both sighted and BVI users. We then used our results to guide the design of a multi-sensory representation of non-visible astronomical data including visual, acoustic, and haptic stimuli. We tested this representation as well, in order to refine and propose it to the public.ResultsThe result is the exhibit “Sense the Universe,” to be used for outreach and education. “Sense the Universe” was part of a museum exhibition attended both by sighted and BVI users, where we collected feedback about the final outcome of our work.DiscussionOur findings suggest the validity of multi-sensory representations for a truly and effective engagement in scientific learning, both in terms of intelligibility and persistence of scientific contents and of a more equal access to scientific culture.

Universal Design Principles for Multimodal Representation in Literacy Activities for Preschoolers

The article demonstrates how to use the multimodal and multisensory representation principle of Universal Design for Learning (UDL) to increase access to storybook reading for diverse groups of preschoolers with extensive support needs (ESN). Storybook reading is an essential part of instruction in early childhood education. Utilizing UDL-inspired book boxes, children of all ability levels can partake in the activity in a meaningful manner. Multimodal representation, such as use of tangible or abstract items (e.g.: photos, miniatures, real items) or sensory experiences (e.g.: essential oils, food items, braille, physical movement) allow children to engage with stories in an enhanced manner. Book boxes can assist in developing concepts related to social emotional growth, social skills, functional living skills, environmental concepts, and vocabulary presented in books.

A neurocomputational analysis of visual bias on bimanual tactile spatial perception during a crossmodal exposure.

Vision and touch both support spatial information processing. These sensory systems also exhibit highly specific interactions in spatial perception, which may reflect multisensory representations that are learned through visuo-tactile (VT) experiences. Recently, Wani and colleagues reported that task-irrelevant visual cues bias tactile perception, in a brightness-dependent manner, on a task requiring participants to detect unimanual and bimanual cues. Importantly, tactile performance remained spatially biased after VT exposure, even when no visual cues were presented. These effects on bimanual touch conceivably reflect cross-modal learning, but the neural substrates that are changed by VT experience are unclear. We previously described a neural network capable of simulating VT spatial interactions. Here, we exploited this model to test different hypotheses regarding potential network-level changes that may underlie the VT learning effects. Simulation results indicated that VT learning effects are inconsistent with plasticity restricted to unisensory visual and tactile hand representations. Similarly, VT learning effects were also inconsistent with changes restricted to the strength of inter-hemispheric inhibitory interactions. Instead, we found that both the hand representations and the inter-hemispheric inhibitory interactions need to be plastic to fully recapitulate VT learning effects. Our results imply that crossmodal learning of bimanual spatial perception involves multiple changes distributed over a VT processing cortical network.

Multisensory representations of space and time in sensory cortices.

Clear evidence demonstrated a supramodal organization of sensory cortices with multisensory processing occurring even at early stages of information encoding. Within this context, early recruitment of sensory areas is necessary for the development of fine domain-specific (i.e., spatial or temporal) skills regardless of the sensory modality involved, with auditory areas playing a crucial role in temporal processing and visual areas in spatial processing. Given the domain-specificity and the multisensory nature of sensory areas, in this study, we hypothesized that preferential domains of representation (i.e., space and time) of visual and auditory cortices are also evident in the early processing of multisensory information. Thus, we measured the event-related potential (ERP) responses of 16 participants while performing multisensory spatial and temporal bisection tasks. Audiovisual stimuli occurred at three different spatial positions and time lags and participants had to evaluate whether the second stimulus was spatially (spatial bisection task) or temporally (temporal bisection task) farther from the first or third audiovisual stimulus. As predicted, the second audiovisual stimulus of both spatial and temporal bisection tasks elicited an early ERP response (time window 50-90 ms) in visual and auditory regions. However, this early ERP component was more substantial in the occipital areas during the spatial bisection task, and in the temporal regions during the temporal bisection task. Overall, these results confirmed the domain specificity of visual and auditory cortices and revealed that this aspect selectively modulates also the cortical activity in response to multisensory stimuli.

Social relevance modulates multisensory integration.

When interacting with the environment, humans exhibit robust biases toward information that pertains to themselves: Self-relevant information is processed faster and yields more accurate responses than information linked to others. Recent studies have shown that simple social associations can lead to the instant deployment of this benefit in the processing of abstract stimuli. However, how self-prioritization evolves across the processing hierarchy has been a subject of intense debate. Furthermore, there is little empirical evidence about the functional efficiency of social relevance in natural environments in which information is present across multiple senses. Across three experiments (each n = 40), the present study shows that self-prioritization effects (a) can arise in simple audio-visual numerosity judgements, (b) can be efficiently deployed across the senses by funneling perception toward self-relevant information in the more reliable sensory modality, and (c) modulate the integration of auditory and visual information into a multisensory representation. Taken together, the present findings suggest that social relevance can influence multisensory processing at both perceptual and postperceptual stages via early attentional modulations of sensory integration and later, task-dependent attentional control. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Multisensory Concept Learning Framework Based on Spiking Neural Networks.

Concept learning highly depends on multisensory integration. In this study, we propose a multisensory concept learning framework based on brain-inspired spiking neural networks to create integrated vectors relying on the concept's perceptual strength of auditory, gustatory, haptic, olfactory, and visual. With different assumptions, two paradigms: Independent Merge (IM) and Associate Merge (AM) are designed in the framework. For testing, we employed eight distinct neural models and three multisensory representation datasets. The experiments show that integrated vectors are closer to human beings than the non-integrated ones. Furthermore, we systematically analyze the similarities and differences between IM and AM paradigms and validate the generality of our framework.

Statistical Analysis of Multisensory and Text-Derived Representations on Concept Learning.

When learning concepts, cognitive psychology research has revealed that there are two types of concept representations in the human brain: language-derived codes and sensory-derived codes. For the objective of human-like artificial intelligence, we expect to provide multisensory and text-derived representations for concepts in AI systems. Psychologists and computer scientists have published lots of datasets for the two kinds of representations, but as far as we know, no systematic work exits to analyze them together. We do a statistical study on them in this work. We want to know if multisensory vectors and text-derived vectors reflect conceptual understanding and if they are complementary in terms of cognition. Four experiments are presented in this work, all focused on multisensory representations labeled by psychologists and text-derived representations generated by computer scientists for concept learning, and the results demonstrate that (1) for the same concept, both forms of representations can properly reflect the concept, but (2) the representational similarity analysis findings reveal that the two types of representations are significantly different, (3) as the concreteness of the concept grows larger, the multisensory representation of the concept becomes closer to human beings than the text-derived representation, and (4) we verified that combining the two improves the concept representation.

Adaptation to delayed visual feedback of the body movement extends multisensory peripersonal space.

The space surrounding the body in which individuals interact with the environment is known as the peripersonal space (PPS). Previous studies have reported that PPS has multisensory nature. However, the relationship between the multisensory nature of PPS and an individuals' defensive actions has not been fully clarified to date. We investigated this relationship by examining the multisensory representation of PPS under situations in which visual feedback of body movements was delayed by using a virtual reality system. The results indicated that body-movement delays extended the multisensory PPS, suggesting that body-movement delays increased the potential threat of distant objects because it was necessary to prepare defensive actions sooner. The previous findings can be interpreted that PPS is modulated by the spatio-temporal relationship between people and external stimuli. This view may provide evidence of interactions between defensive and nondefensive functions of the multisensory PPS.

Semantically Congruent Bimodal Presentation with Divided-Modality Attention Accelerates Unisensory Working Memory Retrieval.

Although previous studies have shown that semantic multisensory integration can be differentially modulated by attention focus, it remains unclear whether attentionally mediated multisensory perceptual facilitation could impact further cognitive performance. Using a delayed matching-to-sample paradigm, the present study investigated the effect of semantically congruent bimodal presentation on subsequent unisensory working memory (WM) performance by manipulating attention focus. The results showed that unisensory WM retrieval was faster in the semantically congruent condition than in the incongruent multisensory encoding condition. However, such a result was only found in the divided-modality attention condition. This result indicates that a robust multisensory representation was constructed during semantically congruent multisensory encoding with divided-modality attention; this representation then accelerated unisensory WM performance, especially auditory WM retrieval. Additionally, an overall faster unisensory WM retrieval was observed under the modality-specific selective attention condition compared with the divided-modality condition, indicating that the division of attention to address two modalities demanded more central executive resources to encode and integrate crossmodal information and to maintain a constructed multisensory representation, leaving few resources for WM retrieval. Additionally, the present finding may support the amodal view that WM has an amodal central storage component that is used to maintain modal-based attention-optimized multisensory representations.

Frontal and parietal background connectivity and their dynamic changes account for individual differences in the multisensory representation of peripersonal space

Functional connectivity (FC) of brain networks dynamically fluctuates during both rest and task execution. Individual differences in dynamic FC have been associated with several cognitive and behavioral traits. However, whether dynamic FC also contributes to sensorimotor representations guiding body-environment interactions, such as the representation of peripersonal space (PPS), is currently unknown. PPS is the space immediately surrounding the body and acts as a multisensory interface between the individual and the environment. We used an audio-tactile task with approaching sounds to map the individual PPS extension, and fMRI to estimate the background FC. Specifically, we analyzed FC values for each stimulus type (near and far space) and its across-trial variability. FC was evaluated between task-relevant nodes of two fronto-parietal networks (the Dorsal Attention Network, DAN, and the Fronto-Parietal Network, FPN) and a key PPS region in the premotor cortex (PM). PM was significantly connected to specific task-relevant nodes of the DAN and the FPN during the audio-tactile task, and FC was stronger while processing near space, as compared to far space. At the individual level, less PPS extension was associated with stronger premotor-parietal FC during processing of near space, while the across-trial variability of premotor-parietal and premotor-frontal FC was higher during the processing of far space. Notably, only across-trial FC variability captured the near-far modulation of space processing. Our findings indicate that PM connectivity with task-relevant frontal and parietal regions and its dynamic changes participate in the mechanisms that enable PPS representation, in agreement with the idea that neural variability plays a crucial role in plastic and dynamic sensorimotor representations.

Multisensory-Guided Associative Learning Enhances Multisensory Representation in Primary Auditory Cortex.

Sensory cortices, classically considered to represent modality-specific sensory information, are also found to engage in multisensory processing. However, how sensory processing in sensory cortices is cross-modally modulated remains an open question. Specifically, we understand little of cross-modal representation in sensory cortices in perceptual tasks and how perceptual learning modifies this process. Here, we recorded neural responses in primary auditory cortex (A1) both while freely moving rats discriminated stimuli in Go/No-Go tasks and when anesthetized. Our data show that cross-modal representation in auditory cortices varies with task contexts. In the task of an audiovisual cue being the target associating with water reward, a significantly higher proportion of auditory neurons showed a visually evoked response. The vast majority of auditory neurons, if processing auditory-visual interactions, exhibit significant multisensory enhancement. However, when the rats performed tasks with unisensory cues being the target, cross-modal inhibition, rather than enhancement, predominated. In addition, multisensory associational learning appeared to leave a trace of plastic change in A1, as a larger proportion of A1 neurons showed multisensory enhancement in anesthesia. These findings indicate that multisensory processing in principle sensory cortices is not static, and having cross-modal interaction in the task requirement can substantially enhance multisensory processing in sensory cortices.

Peripersonal and reaching space differ: Evidence from their spatial extent and multisensory facilitation pattern

Peripersonal space (PPS) is a multisensory representation of the space near body parts facilitating interactions with the close environment. Studies on non-human and human primates agree in showing that PPS is a body part-centered representation that guides actions. Because of these characteristics, growing confusion surrounds peripersonal and arm-reaching space (ARS), that is the space one’s arm can reach. Despite neuroanatomical evidence favoring their distinction, no study has contrasted directly their respective extent and behavioral features. Here, in five experiments (N = 140) we found that PPS differs from ARS, as evidenced both by participants’ spatial and temporal performance and by its modeling. We mapped PPS and ARS using both their respective gold standard tasks and a novel multisensory facilitation paradigm. Results show that: (1) PPS is smaller than ARS; (2) multivariate analyses of spatial patterns of multisensory facilitation predict participants’ hand locations within ARS; and (3) the multisensory facilitation map shifts isomorphically following hand positions, revealing hand-centered coding of PPS, therefore pointing to a functional similarity to the receptive fields of monkeys’ multisensory neurons. A control experiment further corroborated these results and additionally ruled out the orienting of attention as the driving mechanism for the increased multisensory facilitation near the hand. In sharp contrast, ARS mapping results in a larger spatial extent, with undistinguishable patterns across hand positions, cross-validating the conclusion that PPS and ARS are distinct spatial representations. These findings show a need for refinement of theoretical models of PPS, which is relevant to constructs as diverse as self-representation, social interpersonal distance, and motor control.

The Redundant Signals Effect and the Full Body Illusion: not Multisensory, but Unisensory Tactile Stimuli Are Affected by the Illusion.

During a full body illusion (FBI), participants experience a change in self-location towards a body that they see in front of them from a third-person perspective and experience touch to originate from this body. Multisensory integration is thought to underlie this illusion. In the present study we tested the redundant signals effect (RSE) as a new objective measure of the illusion that was designed to directly tap into the multisensory integration underlying the illusion. The illusion was induced by an experimenter who stroked and tapped the participant's shoulder and underarm, while participants perceived the touch on the virtual body in front of them via a head-mounted display. Participants performed a speeded detection task, responding to visual stimuli on the virtual body, to tactile stimuli on the real body and to combined (multisensory) visual and tactile stimuli. Analysis of the RSE with a race model inequality test indicated that multisensory integration took place in both the synchronous and the asynchronous condition. This surprising finding suggests that simultaneous bodily stimuli from different (visual and tactile) modalities will be transiently integrated into a multisensory representation even when no illusion is induced. Furthermore, this finding suggests that the RSE is not a suitable objective measure of body illusions. Interestingly however, responses to the unisensory tactile stimuli in the speeded detection task were found to be slower and had a larger variance in the asynchronous condition than in the synchronous condition. The implications of this finding for the literature on body representations are discussed.

Multisensory integration supports configural learning of a home refuge in the whip spider Phrynus marginemaculatus.

Whip spiders (Amblypygi) reside in structurally complex habitats and are nocturnally active yet display notable navigational abilities. From the theory that uncertainty in sensory inputs should promote multisensory representations to guide behavior, we hypothesized that their navigation is supported by a multisensory and perhaps configural representation of navigational inputs, an ability documented in a few insects and never reported in arachnids. We trained Phrynus marginemaculatus to recognize a home shelter characterized by both discriminative olfactory and tactile stimuli. In tests, subjects readily discriminated between shelters based on the paired stimuli. However, subjects failed to recognize the shelter in tests with either of the component stimuli alone. This result is consistent with the hypothesis that the terminal phase of their navigational behavior, shelter recognition, can be supported by the integration of multisensory stimuli as an enduring, configural representation. We hypothesize that multisensory learning occurs in the whip spiders' extraordinarily large mushroom bodies, which may functionally resemble the hippocampus of vertebrates.

Why There Is a Vestibular Sense, orHowMetacognitionIndividuates the Senses.

Should the vestibular system be counted as a sense? This basic conceptual question remains surprisingly controversial. While it is possible to distinguish specific vestibular organs, it is not clear that this suffices to identify a genuine vestibular sense because of the supposed absence of a distinctive vestibular personal-level manifestation. The vestibular organs instead contribute to more general multisensory representations, whose name still suggest that they have a distinct 'sensory' contribution. The vestibular case shows a good example of the challenge of individuating the senses when multisensory interactions are the norm, neurally, representationally and phenomenally. Here, we propose that an additional metacognitive criterion can be used to single out a distinct sense, besides the existence of specific organs and despite the fact that the information coming from these organs is integrated with other sensory information. We argue that it is possible for human perceivers to monitor information coming from distinct organs, despite their integration, as exhibited and measured through metacognitive performance. Based on the vestibular case, we suggest that metacognitive awareness of the information coming from sensory organs constitutes a new criterion to individuate a sense through both physiological and personal criteria. This new way of individuating the senses accommodates both the specialised nature of sensory receptors as well as the intricate multisensory aspect of neural processes and experience, while maintaining the idea that each sense contributes something special to how we monitor the world and ourselves, at the subjective level.