Multisensory Contexts Research Articles

Unsupervised Characterization of Prediction Error Markers in Unisensory and Multisensory Streams Reveal the Spatiotemporal Hierarchy of Cortical Information Processing.

Elicited upon violation of regularity in stimulus presentation, mismatch negativity (MMN) reflects the brain's ability to perform automatic comparisons between consecutive stimuli and provides an electrophysiological index of sensory error detection whereas P300 is associated with cognitive processes such as updating of the working memory. To date, there has been extensive research on the roles of MMN and P300 individually, because of their potential to be used as clinical markers of consciousness and attention, respectively. Here, we intend to explore with an unsupervised and rigorous source estimation approach, the underlying cortical generators of MMN and P300, in the context of prediction error propagation along the hierarchies of brain information processing in healthy human participants. The existing methods of characterizing the two ERPs involve only approximate estimations of their amplitudes and latencies based on specific sensors of interest. Our objective is twofold: first, we introduce a novel data-driven unsupervised approach to compute latencies and amplitude of ERP components accurately on an individual-subject basis and reconfirm earlier findings. Second, we demonstrate that in multisensory environments, MMN generators seem to reflect a significant overlap of "modality-specific" and "modality-independent" information processing while P300 generators mark a shift toward completely "modality-independent" processing. Advancing earlier understanding that multisensory contexts speed up early sensory processing, our study reveals that temporal facilitation extends to even the later components of prediction error processing, using EEG experiments. Such knowledge can be of value to clinical research for characterizing the key developmental stages of lifespan aging, schizophrenia, and depression.

Cross-Modal Contributions to Episodic Memory for Voices.

Multisensory context often facilitates perception and memory. In fact, encoding items within a multisensory context can improve memory even on strictly unisensory tests (i.e., when the multisensory context is absent). Prior studies that have consistently found these multisensory facilitation effects have largely employed multisensory contexts in which the stimuli were meaningfully related to the items targeting for remembering (e.g., pairing canonical sounds and images). Other studies have used unrelated stimuli as multisensory context. A third possible type of multisensory context is one that is environmentally related simply because the stimuli are often encountered together in the real world. We predicted that encountering such a multisensory context would also enhance memory through cross-modal associations, or representations relating to one's prior multisensory experience with that sort of stimuli in general. In two memory experiments, we used faces and voices of unfamiliar people as everyday stimuli individuals have substantial experience integrating the perceptual features of. We assigned participants to face- or voice-recognition groups and ensured that, during the study phase, half of the face or voice targets were encountered also with information in the other modality. Voices initially encoded along with faces were consistently remembered better, providing evidence that cross-modal associations could explain the observed multisensory facilitation.

The influence of multisensory indoor environment on the perception of orange juice

Drinking (as well as eating) represents a fundamental multisensory experience of our lives. Recent studies have proposed the implementation of multisensory contexts to improve the taste perception of food and beverages. This topic becomes relevant when considering the possibility of promoting the consumption of healthy beverages (e.g., orange juice). Only a few studies have concentrated on the multisensory features of the indoor environment (e.g., acoustic and visual context) reporting effects on the perception of healthy beverages. This study investigates how multisensory indoor environments modify the taste perception while drinking orange juice. By controlling the room microclimate and combining the correlated colour temperature of lighting (i.e. warm and cool) and the background wall colour (i.e., red, green) in the Sens i-Lab, with different audio stimuli (i.e. low and high-frequency pure tone), participants were invited to taste the low-sugar orange juice and to rate the fruit juice aroma, sweetness, sourness, bitterness, freshness, and thickness. The results showed that listening to a high-frequency tone in an environment with warm lighting and a red background increases the sweetness perception of orange juice compared to the control environment. In addition, compared to the control condition, the juice aroma was rated more intense in almost all multisensory conditions and thicker with cool lighting and a green background while listening to a low-frequency tone. Our findings suggested that the development of multisensory design can contribute to interior design strategies and design innovation in food architecture, where people can improve their taste in healthy foods and beverages.

S1 represents multisensory contexts and somatotopic locations within and outside the bounds of the cortical homunculus

Recent literature suggests that tactile events are represented in the primary somatosensory cortex (S1) beyond its long-established topography; in addition, the extent to which S1 is modulated by vision remains unclear. To better characterize S1, human electrophysiological data were recorded during touches to the forearm or finger. Conditions included visually observed physical touches, physical touches without vision, and visual touches without physical contact. Two major findings emerge from this dataset. First, vision strongly modulates S1 area 1, but only if there is a physical element to the touch, suggesting that passive touch observation is insufficient to elicit neural responses. Second, despite recording in a putative arm area of S1, neural activity represents both arm and finger stimuli during physical touches. Arm touches are encoded more strongly and specifically, supporting the idea that S1 encodes tactile events primarily through its topographic organization but also more generally, encompassing other areas of the body.

Perception of temporal cues in word segments under bimodal divided attention in simulated cochlear implant listening

Auditory temporal processing is important for cochlear-implant (CI) users to understand speech. Speech processing often unfolds along with sensory inputs from other modalities. This study examined whether dividing attention across modalities affects temporal processing of word segments in simulated CI listening. Adults with normal hearing performed categorization tasks on unprocessed and 8-channel vocoded versions of a Dish-Ditch contrast with varying silent intervals preceding the word-final fricative. The categorization performance was measured concurrently with a visual working memory task on meaningless images. This divided attention (AV-div) condition was compared with an audiovisual selective attention (AV-sel) condition in which participants focused on the categorization task and ignored the visual stimuli, and with an auditory-only (AO) condition in which participants performed the categorization task without the visual stimuli. Preliminary results (N = 5) revealed that participants tended to report more “Dish” responses for words with longer silence intervals under the two AV conditions than the AO condition, especially for the AV-div condition. The AV-related effects on the categorization performance appear to be reduced for vocoded than unprocessed words. These results suggest that concurrent visual information affects auditory temporal processing, which might lead to speech understanding difficulties of CI users in multisensory contexts.

EEG Data Quality in Real-World Settings: Examining Neural Correlates of Attention in School-Aged Children.

Advances in mobile electroencephalography (EEG) technology have made it possible to examine covert cognitive processes in real-world settings such as student attention in the classroom. Here, we outline research using wired and wireless EEG technology to examine attention in elementary school children across increasingly naturalistic paradigms in schools, ranging from a lab-based paradigm where children met one-on-one with an experimenter in a field laboratory to mobile EEG testing conducted in the same school during semi-naturalistic classroom lessons. Despite an increase of data loss with the classroom-based paradigm, we demonstrate that it is feasible to collect quality data in classroom settings with young children. We also provide a test case for how robust EEG signals, such as alpha oscillations, can be used to identify measurable differences in covert processes like attention in classrooms. We end with pragmatic suggestions for researchers interested in employing naturalistic EEG methods in real-world, multisensory contexts.

IoT-Based Patient Health Data Using Improved Context-Aware Data Fusion and Enhanced Recursive Feature Elimination Model

The Internet of Things (IoT) in the healthcare market is propelled forward by the implementation of digital systems for monitoring and analysing health problems. IoT and smart devices can contribute to a highly smart environment. Smart medical devices interconnected with smartphone apps can collect medical and other required health data. "Data Fusion (DF)" refers to integrating data and knowledge from multiple sources. However, these techniques are also applied to other domains, including text processing. Using data from multiple distributed sources, the objective of DF in multisensory contexts is to reduce the chance of detection errors and increase their reliability. The objective is to increase scalability, performance efficiency, and identification. A medical device’s ability to scale up or down demonstrates its capacity to respond to environmental factors. A more scalable system performs as expected, with no interruptions, and makes the best available use of the resource management it has. To ensure that these tracking devices all work the same way, it is essential to form a specialised group to develop uniformity in areas such as communication channels, aggregation of data, and smart interfaces. The main contribution of this research is pre-processing, DF using the Improved Context-aware Data Fusion (ICDF) algorithm, feature extraction via Improved Principal Component Analysis (IPCA), feature selection through the Enhanced Recursive Feature Elimination (ERFE) algorithm, and a classifier using an ensemble-based Machine Learning (ML) model. The Improved Dynamic Bayesian Network (IDBN) is a good trade-off for tractability, becoming a tool for ICDF operations. The simulation results show that the proposed ICDF model achieves higher performance in terms of 97% accuracy, 96% precision, 97% recall, and 97% F1 score in the healthcare system.

Selective adaptation in speech: Measuring the effects of visual and lexical contexts.

Speech selective adaptation is a phenomenon in which repeated presentation of a speech stimulus alters subsequent phonetic categorization. Prior work has reported that lexical, but not multisensory, context influences selective adaptation. This dissociation suggests that lexical and multisensory contexts influence speech perception through separate and independent processes (see Samuel & Lieblich, 2014). However, this dissociation is based on results reported by different studies using different stimuli. This leaves open the possibility that the divergent effects of multisensory and lexical contexts on selective adaptation may be the result of idiosyncratic differences in the stimuli rather than separate perceptual processes. The present investigation used a single stimulus set to compare the selective adaptation produced by lexical and multisensory contexts. In contrast to the apparent dissociation in the literature, we find that multisensory information can in fact support selective adaptation. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Minimal interplay between explicit knowledge, dynamics of learning and temporal expectations in different, complex uni- and multisensory contexts

While temporal expectations (TE) generally improve reactions to temporally predictable events, it remains unknown how the learning of temporal regularities (one time point more likely than another time point) and explicit knowledge about temporal regularities contribute to performance improvements; and whether any contributions generalise across modalities. Here, participants discriminated the frequency of diverging auditory, visual or audio-visual targets embedded in auditory, visual or audio-visual distractor sequences. Temporal regularities were manipulated run-wise (early vs. late target within sequence). Behavioural performance (accuracy, RT) plus measures from a computational learning model all suggest that learning of temporal regularities occurred but did not generalise across modalities, and that dynamics of learning (size of TE effect across runs) and explicit knowledge have little to no effect on the strength of TE. Remarkably, explicit knowledge affects performance—if at all—in a context-dependent manner: Only under complex task regimes (here, unknown target modality) might it partially help to resolve response conflict while it is lowering performance in less complex environments.

Developing of Focal Ischemia in the Hippocampus or the Amygdala Reveals a Regional Compensation Rule for Fear Memory Acquisition.

Circuit compensation is often observed in patients with acute ischemic stroke, suggesting the importance of the interaction between brain regions. Also, contextual fear memory is an association between multisensory contexts and fearful stimuli, for which the interaction between the hippocampus and the amygdala is believed to be critical. To understand how focal ischemia in one region could influence the other region, we used a modified photo-thrombosis to induce focal ischemia in the hippocampus or the amygdala or both in freely-moving rats. We found that the learning curve and short-term memory (STM) were not affected in the rats although focal ischemia was induced 5 h before learning in either the hippocampus or the amygdala; these were impaired by the induction of ischemia in both the regions. Furthermore, the learning curve and STM were impaired when ischemia was induced 24 h before learning in either the hippocampus or the amygdala when the synaptic transmission was altered in one region because of ischemia in the other region. These results suggest that the circuit compensation between the hippocampus and the amygdala is critical for fear memory acquisition.

Towards more-than-human digital data studies: developing research-creation methods

As the number of digital technologies expands, entering more domains of everyday life, people’s activities, bodies and preferences are rendered into constantly changing flows of digitised information. The interdisciplinary field of critical data studies has emerged in response. In this article, we outline the design and development of methods employed in our new project ‘Living with Personal Data’ as a move towards expanding the knowledge base and methodological approaches of critical data studies. Our approach takes up more-than-human theoretical perspectives and research-creation methods to elicit the affective and multisensory contexts of people’s feelings, practices and imaginaries concerning their digital data. We describe a set of workshops established to experiment with some new methods we have devised for our project’s fieldwork. The article ends with some reflections on what these theories and methods can offer for a reimagined digital data studies that can acknowledge and surface more-than-human dimensions.

Pure tones modulate the representation of orientation and direction in the primary visual cortex.

Multimodal sensory integration facilitates the generation of a unified and coherent perception of the environment. It is now well established that unimodal sensory perceptions, such as vision, are improved in multisensory contexts. Whereas multimodal integration is primarily performed by dedicated multisensory brain regions such as the association cortices or the superior colliculus, recent studies have shown that multisensory interactions also occur in primary sensory cortices. In particular, sounds were shown to modulate the responses of neurons located in layers 2/3 (L2/3) of the mouse primary visual cortex (V1). Yet, the net effect of sound modulation at the V1 population level remained unclear. In the present study, we performed two-photon calcium imaging in awake mice to compare the representation of the orientation and the direction of drifting gratings by V1 L2/3 neurons in unimodal (visual only) or multimodal (audiovisual) conditions. We found that sound modulation depended on the tuning properties (orientation and direction selectivity) and response amplitudes of V1 L2/3 neurons. Sounds potentiated the responses of neurons that were highly tuned to the cue's orientation and direction but weakly active in the unimodal context, following the principle of inverse effectiveness of multimodal integration. Moreover, sound suppressed the responses of neurons untuned for the orientation and/or the direction of the visual cue. Altogether, sound modulation improved the representation of the orientation and direction of the visual stimulus in V1 L2/3. Namely, visual stimuli presented with auditory stimuli recruited a neuronal population better tuned to the visual stimulus orientation and direction than when presented alone. NEW & NOTEWORTHY The primary visual cortex (V1) receives direct inputs from the primary auditory cortex. Yet, the impact of sounds on visual processing in V1 remains controverted. We show that the modulation by pure tones of V1 visual responses depends on the orientation selectivity, direction selectivity, and response amplitudes of V1 neurons. Hence, audiovisual stimuli recruit a population of V1 neurons better tuned to the orientation and direction of the visual stimulus than unimodal visual stimuli.

Variance adaptation in navigational decision making.

Sensory systems relay information about the world to the brain, which enacts behaviors through motor outputs. To maximize information transmission, sensory systems discard redundant information through adaptation to the mean and variance of the environment. The behavioral consequences of sensory adaptation to environmental variance have been largely unexplored. Here, we study how larval fruit flies adapt sensory-motor computations underlying navigation to changes in the variance of visual and olfactory inputs. We show that variance adaptation can be characterized by rescaling of the sensory input and that for both visual and olfactory inputs, the temporal dynamics of adaptation are consistent with optimal variance estimation. In multisensory contexts, larvae adapt independently to variance in each sense, and portions of the navigational pathway encoding mixed odor and light signals are also capable of variance adaptation. Our results suggest multiplication as a mechanism for odor-light integration.

Phasic and sustained interactions of multisensory interplay and temporal expectation

Every moment organisms are confronted with complex streams of information which they use to generate a reliable mental model of the world. There is converging evidence for several optimization mechanisms instrumental in integrating (or segregating) incoming information; among them are multisensory interplay (MSI) and temporal expectation (TE). Both mechanisms can account for enhanced perceptual sensitivity and are well studied in isolation; how these two mechanisms interact is currently less well-known. Here, we tested in a series of four psychophysical experiments for TE effects in uni- and multisensory contexts with different levels of modality-related and spatial uncertainty. We found that TE enhanced perceptual sensitivity for the multisensory relative to the best unisensory condition (i.e. multisensory facilitation according to the max-criterion). In the latter TE effects even vanished if stimulus-related spatial uncertainty was increased. Accordingly, computational modelling indicated that TE, modality-related and spatial uncertainty predict multisensory facilitation. Finally, the analysis of stimulus history revealed that matching expectation at trial n-1 selectively improves multisensory performance irrespective of stimulus-related uncertainty. Together, our results indicate that benefits of multisensory stimulation are enhanced by TE especially in noisy environments, which allows for more robust information extraction to boost performance on both short and sustained time ranges.

A Critical Role for Thermosensation in Host Seeking by Skin-Penetrating Nematodes

Skin-penetrating parasitic nematodes infect approximately one billion people worldwide and are a major source of neglected tropical disease [1-6]. Their life cycle includes an infective third-larval (iL3) stage that searches for hosts to infect in a poorly understood process that involves both thermal and olfactory cues. Here, we investigate the temperature-driven behaviors of skin-penetrating iL3s, including the human-parasitic threadworm Strongyloides stercoralis and the human-parasitic hookworm Ancylostoma ceylanicum. We show that human-parasitic iL3s respond robustly to thermal gradients. Like the free-living nematode Caenorhabditis elegans, human-parasitic iL3s show both positive and negative thermotaxis, and the switch between them is regulated by recent cultivation temperature [7]. When engaging in positive thermotaxis, iL3s migrate toward temperatures approximating mammalian body temperature. Exposing iL3s to a new cultivation temperature alters the thermal switch point between positive and negative thermotaxis within hours, similar to the timescale of thermal plasticity in C.elegans [7]. Thermal plasticity in iL3s may enable them to optimize host finding on a diurnal temperature cycle. We show that temperature-driven responses can be dominant in multisensory contexts such that, when thermal drive is strong, iL3s preferentially engage in temperature-driven behaviors despite the presence of an attractive host odorant. Finally, targeted mutagenesis of the S.stercoralis tax-4 homolog abolishes heat seeking, providing the first evidence that parasitic host-seeking behaviors are generated through an adaptation of sensory cascades that drive environmental navigation in C.elegans [7-10]. Together, our results provide insight into the behavioral strategies and molecular mechanisms that allow skin-penetrating nematodes to target humans.

Colorful insights supporting the modeling of creative processes across language, music and emotion

In an attempt to unite advances in color science, the creative arts, cognitive psychology and engineering to foster an improved understanding of creative processes, we elaborate on and discuss some recent findings across creative art experiments, cognitive psychology and perceptual color compression. Motivated by a fundamental phenomenon of efficiently managing functional complexity, which has been observed, for example, in design tool development as well as product development in microelectronics, we propose associating functional structures of language and music with perceptual sensitivity to hue, simultaneous color contrast, and color saturation across high dynamic range color processing while simultaneously considering elementary models used in quantum mechanics. A multi-dimensional functional hierarchy not only reveals similarities but also offers a platform towards open questions about creative thinking processes which can be more easily elaborated in a distributive differential context provided by a dynamic spatio-temporal (space-time) model. A suitable test scenario emerges from forward-looking creative experiments utilizing the arts in combination with a highly generic perceptual color compression model. A well engineered verification toolset based on nonlinear control theory appears useful in addressing fundamental challenges in highly complex neuronal tasks across language, music and emotion. We also focus on interfacing between terminologies of cognitive, mathematical and engineering domains together with their underlying functionalities. We present preliminary thoughts and discuss our unconventionally structured approach in multi-modal and multi-sensory contexts via comparison of similarities as well as differences between creativity tasks and perceptual color compression. Ongoing work shall focus specifically on elaborating effective strategies that combine the proposed functional models with promising Deep Learning architectures.

The role of multisensory interplay in enabling temporal expectations

Temporal regularities can guide our attention to focus on a particular moment in time and to be especially vigilant just then. Previous research provided evidence for the influence of temporal expectation on perceptual processing in unisensory auditory, visual, and tactile contexts. However, in real life we are often exposed to a complex and continuous stream of multisensory events. Here we tested – in a series of experiments – whether temporal expectations can enhance perception in multisensory contexts and whether this enhancement differs from enhancements in unisensory contexts. Our discrimination paradigm contained near-threshold targets (subject-specific 75% discrimination accuracy) embedded in a sequence of distractors. The likelihood of target occurrence (early or late) was manipulated block-wise. Furthermore, we tested whether spatial and modality-specific target uncertainty (i.e. predictable vs. unpredictable target position or modality) would affect temporal expectation (TE) measured with perceptual sensitivity (d′) and response times (RT). In all our experiments, hidden temporal regularities improved performance for expected multisensory targets. Moreover, multisensory performance was unaffected by spatial and modality-specific uncertainty, whereas unisensory TE effects on d′ but not RT were modulated by spatial and modality-specific uncertainty. Additionally, the size of the temporal expectation effect, i.e. the increase in perceptual sensitivity and decrease of RT, scaled linearly with the likelihood of expected targets. Finally, temporal expectation effects were unaffected by varying target position within the stream. Together, our results strongly suggest that participants quickly adapt to novel temporal contexts, that they benefit from multisensory (relative to unisensory) stimulation and that multisensory benefits are maximal if the stimulus-driven uncertainty is highest. We propose that enhanced informational content (i.e. multisensory stimulation) enables the robust extraction of temporal regularities which in turn boost (uni-)sensory representations.

A multisensory perspective on object memory

Traditional studies of memory and object recognition involved objects presented within a single sensory modality (i.e., purely visual or purely auditory objects). However, in naturalistic settings, objects are often evaluated and processed in a multisensory manner. This begets the question of how object representations that combine information from the different senses are created and utilised by memory functions. Here we review research that has demonstrated that a single multisensory exposure can influence memory for both visual and auditory objects. In an old/new object discrimination task, objects that were presented initially with a task-irrelevant stimulus in another sense were better remembered compared to stimuli presented alone, most notably when the two stimuli were semantically congruent. The brain discriminates between these two types of object representations within the first 100ms post-stimulus onset, indicating early “tagging” of objects/events by the brain based on the nature of their initial presentation context. Interestingly, the specific brain networks supporting the improved object recognition vary based on a variety of factors, including the effectiveness of the initial multisensory presentation and the sense that is task-relevant. We specify the requisite conditions for multisensory contexts to improve object discrimination following single exposures, and the individual differences that exist with respect to these improvements. Our results shed light onto how memory operates on the multisensory nature of object representations as well as how the brain stores and retrieves memories of objects.

It is better when expected: aligning speech and motor rhythms enhances verbal processing

ABSTRACTRhythm is a powerful way to shape the processing of complex sounds such as speech or music by generating temporal expectancies in the listener. Here, we investigated if multisensory expectancies generated by aligning speech and motor rhythms may enhance verbal processing. Participants listened to rhythmically regular German sentences and detected word changes occurring on stressed or unstressed syllables. Participants were cued to produce finger taps simultaneously with the auditory speech rhythm. Finger taps were aligned or misaligned with stressed syllables. Detection of word changes was facilitated when manual movements were temporally aligned with the auditory speech rhythm. Moreover, motor alignment enhanced sensitivity to detect changes on stressed syllables compared to a perceptual control condition. Thus, rhythmic speech structure reinforced by concurrent movement in multisensory contexts has beneficial effects on verbal processing. This finding lends support to models of expectancy-driven speech processing.

Memory and learning with rapid audiovisual sequences.

We examined short-term memory for sequences of visual stimuli embedded in varying multisensory contexts. In two experiments, subjects judged the structure of the visual sequences while disregarding concurrent, but task-irrelevant auditory sequences. Stimuli were eight-item sequences in which varying luminances and frequencies were presented concurrently and rapidly (at 8 Hz). Subjects judged whether the final four items in a visual sequence identically replicated the first four items. Luminances and frequencies in each sequence were either perceptually correlated (Congruent) or were unrelated to one another (Incongruent). Experiment 1 showed that, despite encouragement to ignore the auditory stream, subjects' categorization of visual sequences was strongly influenced by the accompanying auditory sequences. Moreover, this influence tracked the similarity between a stimulus's separate audio and visual sequences, demonstrating that task-irrelevant auditory sequences underwent a considerable degree of processing. Using a variant of Hebb's repetition design, Experiment 2 compared musically trained subjects and subjects who had little or no musical training on the same task as used in Experiment 1. Test sequences included some that intermittently and randomly recurred, which produced better performance than sequences that were generated anew for each trial. The auditory component of a recurring audiovisual sequence influenced musically trained subjects more than it did other subjects. This result demonstrates that stimulus-selective, task-irrelevant learning of sequences can occur even when such learning is an incidental by-product of the task being performed.