Multisensory Context Research Articles

Multisensory object perception in infancy: 4-month-olds perceive a mistuned harmonic as a separate auditory and visual object

Infants learn to use auditory and visual information to organize the sensory world into identifiable objects with particular locations. Here we use a behavioural method to examine infants’ use of harmonicity cues to auditory object perception in a multisensory context. Sounds emitted by different objects sum in the air and the auditory system must figure out which parts of the complex waveform belong to different sources (auditory objects). One important cue to this source separation is that complex tones with pitch typically contain a fundamental frequency and harmonics at integer multiples of the fundamental. Consequently, adults hear a mistuned harmonic in a complex sound as a distinct auditory object (Alain, Theunissen, Chevalier, Batty, & Taylor, 2003). Previous work by our group demonstrated that 4-month-old infants are also sensitive to this cue. They behaviourally discriminate a complex tone with a mistuned harmonic from the same complex with in-tune harmonics, and show an object-related event-related potential (ERP) electrophysiological (EEG) response to the stimulus with mistuned harmonics. In the present study we use an audiovisual procedure to investigate whether infants perceive a complex tone with an 8% mistuned harmonic as emanating from two objects, rather than merely detecting the mistuned cue. We paired in-tune and mistuned complex tones with visual displays that contained either one or two bouncing balls. Four-month-old infants showed surprise at the incongruous pairings, looking longer at the display of two balls when paired with the in-tune complex and at the display of one ball when paired with the mistuned harmonic complex. We conclude that infants use harmonicity as a cue for source separation when integrating auditory and visual information in object perception.

When preschoolers follow their eyes and older children follow their noses: visuo-olfactory social affective matching inchildhood.

Recognition of emotional facial expressions is a crucial skill for adaptive behavior that most often occurs in a multi-sensory context. Affective matching tasks have been used across development to investigate how people integrate facial information with other senses. Given the relative affective strength of olfaction and its relevance in mediating social information since birth, we assessed olfactory-visual matching abilities in a group of 140 children between the ages of 3 and 11 years old. We presented one of three odor primes (rose, fish and no-odor, rated as pleasant or unpleasant by individual children) before a facial choice task (happy vs. disgusted face). Children were instructed to select one of two faces. As expected, children of all ages tended to choose happy faces. Children younger than 5 years of age were biased towards choosing the happy face, irrespective of the odor smelled. After age 5, an affective matching strategy guided children's choices. Smelling a pleasant odor predicted the choice of happy faces, whereas smelling the unpleasant or fish odor predicted the choice of disgusted faces. The present study fills a gap in the developmental literature on olfactory-visual affective strategies that affect decision-making, and represents an important step towards understanding the underlying developmental processes that shape the typical social mind.

Investigating Expressive Tactile Interaction Design in Artistic Graphical Representations

In this article, a design research approach is taken to investigate expressive design of tactile interactions. Most research efforts to date on designing and exploring the representational aspects of tactile interfaces have focused on usability and task-oriented scenarios. Yet, there is limited knowledge on how to aid the design of tactile interfaces that support the design of expressive or user-experience-oriented tactile interactions. We address this gap by studying tactile designs in a multisensory context, where the tactile interface augments works of visual art. The expressive and artistic context introduces new opportunities to extend on previous work, and identify new design and interaction potentials with tactile interfaces in graphical multisensory scenarios. During one-on-one guided design sessions, visual artists were asked to create tactile design prototypes that augmented one of their existing works. Each element of the overall tactile design, regarded as atactile feature, was analyzed using both the bottom-up and top-down approaches. The results discovered through grounded theory are presented and discussed with respect to semiotic theory. Accordingly,tactile constructsandtactile intentsdefine the “form” and “meaning” components of each tactile feature, respectively. Overall analysis of the findings indicates associations among the identified categories and between the two components, leading to design implications for expressive tactile interfaces. Insights from thetactile intentssuggest a set of affordances for expressive visuotactile interactions, which we introduce under the notion ofexpressive roles. Additionally, implications from thetactile constructsindicate a design space for an expressive tactile augmentation design tool, based on which a user interface architecture is proposed. Findings from this research can assist in developing systems and tools for expressive tactile interface design and inspire research in user experience and behavior in multisensory tactile interaction scenarios.

Animal crackers in my … book? Effects of shared reading on parents’ memory for product placement in children’s books

ABSTRACTThe unique combination of characteristics that contribute to the shared reading of children’s picture books foster involvement, engagement, and communication, and result in socialization and development. They also have the potential to make readers susceptible to product placements used in the medium. Currently research on product placement spans almost every medium, with the notable exception of children’s picture books, where research on the practice has been underserved despite its existing presence. The present study recognizes the combination of multi-sensory and social context encoding that takes place during shared reading in this medium and explores how cognitive busyness arising from parent-child engagement with the narrative may inhibit persuasion knowledge while reading children’s books. Using a laboratory experiment and quantitative statistical analysis, the present study demonstrates that shared reading positively affects parents’ recollection of product placements in children’s picture books. It also discusses theoretical and practical implications of product placement recollection.

Seeing Circles and Drawing Ellipses: When Sound Biases Reproduction of Visual Motion.

The perception and production of biological movements is characterized by the 1/3 power law, a relation linking the curvature and the velocity of an intended action. In particular, motions are perceived and reproduced distorted when their kinematics deviate from this biological law. Whereas most studies dealing with this perceptual-motor relation focused on visual or kinaesthetic modalities in a unimodal context, in this paper we show that auditory dynamics strikingly biases visuomotor processes. Biologically consistent or inconsistent circular visual motions were used in combination with circular or elliptical auditory motions. Auditory motions were synthesized friction sounds mimicking those produced by the friction of the pen on a paper when someone is drawing. Sounds were presented diotically and the auditory motion velocity was evoked through the friction sound timbre variations without any spatial cues. Remarkably, when subjects were asked to reproduce circular visual motion while listening to sounds that evoked elliptical kinematics without seeing their hand, they drew elliptical shapes. Moreover, distortion induced by inconsistent elliptical kinematics in both visual and auditory modalities added up linearly. These results bring to light the substantial role of auditory dynamics in the visuo-motor coupling in a multisensory context.

In-situ data collection for oil palm tree height determination using synthetic aperture radar

The oil palm is recognized as the “golden crop,” producing the highest oil yield among oil seed crops. Malaysia, the world's second largest producer of palm oil, has 16 per cent of its territory planted with oil palms. To cope with the increasing global demand on edible oil, additional areas of oil palm are forecast to increase globally by 12 to 19 million hectares by 2050. Due to the limited land bank in Malaysia, new strategies have to be developed to avoid unauthorized clearing of primary forest for the use of oil palm cultivation. Microwave remote sensing could play a part by providing relevant, timely and accurate information for a plantation monitoring system. The use of synthetic aperture radar (SAR) has the advantage of daylight- and weather-independence, a criterion that is very relevant in constantly cloud-covered tropical regions, such as Malaysia. Using interferometric SAR, (InSAR) topographical and tree height profiles of oil palm plantations can be created; such information is useful for mapping oil palm age profiles of the plantations in the country. This paper reports on the use of SAR and InSAR in a multisensory context to provide up-to-date information at plantation level. Remote sensing and in-situ data collection for tree height determination are described. Further research to be carried out over the next two years is outlined.

The Neurobiology Shaping Affective Touch: Expectation, Motivation, and Meaning in the Multisensory Context.

Inter-individual touch can be a desirable reward that can both relieve negative affect and evoke strong feelings of pleasure. However, if other sensory cues indicate it is undesirable to interact with the toucher, the affective experience of the same touch may be flipped to disgust. While a broad literature has addressed, on one hand the neurophysiological basis of ascending touch pathways, and on the other hand the central neurochemistry involved in touch behaviors, investigations of how external context and internal state shapes the hedonic value of touch have only recently emerged. Here, we review the psychological and neurobiological mechanisms responsible for the integration of tactile “bottom–up” stimuli and “top–down” information into affective touch experiences. We highlight the reciprocal influences between gentle touch and contextual information, and consider how, and at which levels of neural processing, top-down influences may modulate ascending touch signals. Finally, we discuss the central neurochemistry, specifically the μ-opioids and oxytocin systems, involved in affective touch processing, and how the functions of these neurotransmitters largely depend on the context and motivational state of the individual.

Crossmodal processing and sensory substitution: Is "seeing" with sound and touch a form of perception or cognition?

The brain has evolved in this multisensory context to perceive the world in an integrated fashion. Although there are good reasons to be skeptical of the influence of cognition on perception, here we argue that the study of sensory substitution devices might reveal that perception and cognition are not necessarily distinct, but rather continuous aspects of our information processing capacities.

A conceptual framework for the assessment of brand congruent sensory modalities

Companies wishing to engage in multisensory marketing by adding previously unused sensory modalities to their brands need to ask how their brand identity translates into these unoccupied modalities. Identifying brand identity congruent sensory modalities in a unisensory fashion is not helpful as multisensory interaction effects change their meaning. Furthermore, existing sensory evaluation techniques do not meet the requirements of marketing managers to serve this purpose. This article addresses this gap by first analysing a multisensory marketing strategy of adding previously unused sensory modalities to an existing brand. Theory supports this strategy, because it enhances the recognition, evaluation and memory of the brand and lets the brand profit from multisensory enhancement. Paramount for this strategy to work is to guarantee the congruency between the newly added sensory modalities with the existing ones of the brand and the brand personality. Therefore, as a second goal, this article proposes a conceptual framework to assess the congruency of sensory modalities in a multisensory semantic context. A three-step process is presented that draws on evaluation techniques used to assess congruency in psychology and the sensory evaluation of food.

Predictive coding and multisensory integration: an attentional account of the multisensory mind.

Multisensory integration involves a host of different cognitive processes, occurring at different stages of sensory processing. Here I argue that, despite recent insights suggesting that multisensory interactions can occur at very early latencies, the actual integration of individual sensory traces into an internally consistent mental representation is dependent on both top–down and bottom–up processes. Moreover, I argue that this integration is not limited to just sensory inputs, but that internal cognitive processes also shape the resulting mental representation. Studies showing that memory recall is affected by the initial multisensory context in which the stimuli were presented will be discussed, as well as several studies showing that mental imagery can affect multisensory illusions. This empirical evidence will be discussed from a predictive coding perspective, in which a central top–down attentional process is proposed to play a central role in coordinating the integration of all these inputs into a coherent mental representation.

Oscillatory mechanisms underlying the enhancement of visual motion perception by multisensory congruency

Multisensory interactions shape every day perception and stimuli in one modality can enhance perception in another even when not being directly task relevant. While the underlying neural principles are slowly becoming evident, most work has focused on transient stimuli and little is known about those mechanisms underlying audio–visual motion processing. We studied the facilitation of visual motion perception by auxiliary sounds, i.e. sounds that by themselves do not provide the specific evidence required for the perceptual task at hand. In our experiment human observers became significantly better at detecting visual random dot motion when this was accompanied by auxiliary acoustic motion rather than stationary sounds. EEG measurements revealed that both auditory and visual motion modulated low frequency oscillations over the respective sensory cortices. Using single trial decoding we quantified those oscillatory signatures permitting the discrimination of visual motion similar to the subject's task. This revealed visual motion-related signatures in low (1–4Hz) and alpha (8–12Hz) bands that were significantly enhanced during congruent compared to disparate audio–visual conditions. Importantly, the auditory enhancement of these oscillatory signatures was predictive of the perceptual multisensory facilitation. These findings emphasise the importance of slow and alpha rhythms for perception in a multisensory context and suggest that acoustic motion can enhance visual perception by means of attention or priming-related mechanisms that are reflected in rhythmic activity over parieto-occipital regions.

Recalibration of the Multisensory Temporal Window of Integration Results from Changing Task Demands

The notion of the temporal window of integration, when applied in a multisensory context, refers to the breadth of the interval across which the brain perceives two stimuli from different sensory modalities as synchronous. It maintains a unitary perception of multisensory events despite physical and biophysical timing differences between the senses. The boundaries of the window can be influenced by attention and past sensory experience. Here we examined whether task demands could also influence the multisensory temporal window of integration. We varied the stimulus onset asynchrony between simple, short-lasting auditory and visual stimuli while participants performed two tasks in separate blocks: a temporal order judgment task that required the discrimination of subtle auditory-visual asynchronies, and a reaction time task to the first incoming stimulus irrespective of its sensory modality. We defined the temporal window of integration as the range of stimulus onset asynchronies where performance was below 75% in the temporal order judgment task, as well as the range of stimulus onset asynchronies where responses showed multisensory facilitation (race model violation) in the reaction time task. In 5 of 11 participants, we observed audio-visual stimulus onset asynchronies where reaction time was significantly accelerated (indicating successful integration in this task) while performance was accurate in the temporal order judgment task (indicating successful segregation in that task). This dissociation suggests that in some participants, the boundaries of the temporal window of integration can adaptively recalibrate in order to optimize performance according to specific task demands.

Cross-Modal Stimulus Conflict: The Behavioral Effects of Stimulus Input Timing in a Visual-Auditory Stroop Task

Cross-modal processing depends strongly on the compatibility between different sensory inputs, the relative timing of their arrival to brain processing components, and on how attention is allocated. In this behavioral study, we employed a cross-modal audio-visual Stroop task in which we manipulated the within-trial stimulus-onset-asynchronies (SOAs) of the stimulus-component inputs, the grouping of the SOAs (blocked vs. random), the attended modality (auditory or visual), and the congruency of the Stroop color-word stimuli (congruent, incongruent, neutral) to assess how these factors interact within a multisensory context. One main result was that visual distractors produced larger incongruency effects on auditory targets than vice versa. Moreover, as revealed by both overall shorter response times (RTs) and relative shifts in the psychometric incongruency-effect functions, visual-information processing was faster and produced stronger and longer-lasting incongruency effects than did auditory. When attending to either modality, stimulus incongruency from the other modality interacted with SOA, yielding larger effects when the irrelevant distractor occurred prior to the attended target, but no interaction with SOA grouping. Finally, relative to neutral-stimuli, and across the wide range of the SOAs employed, congruency led to substantially more behavioral facilitation than did incongruency to interference, in contrast to findings that within-modality stimulus-compatibility effects tend to be more evenly split between facilitation and interference. In sum, the present findings reveal several key characteristics of how we process the stimulus compatibility of cross-modal sensory inputs, reflecting stimulus processing patterns that are critical for successfully navigating our complex multisensory world.

The Efficacy of Single-Trial Multisensory Memories

This review article summarizes evidence that multisensory experiences at one point in time have long-lasting effects on subsequent unisensory visual and auditory object recognition. The efficacy of single-trial exposure to task-irrelevant multisensory events is its ability to modulate memory performance and brain activity to unisensory components of these events presented later in time. Object recognition (either visual or auditory) is enhanced if the initial multisensory experience had been semantically congruent and can be impaired if this multisensory pairing was either semantically incongruent or entailed meaningless information in the task-irrelevant modality, when compared to objects encountered exclusively in a unisensory context. Processes active during encoding cannot straightforwardly explain these effects; performance on all initial presentations was indistinguishable despite leading to opposing effects with stimulus repetitions. Brain responses to unisensory stimulus repetitions differ during early processing stages (-100 ms post-stimulus onset) according to whether or not they had been initially paired in a multisensory context. Plus, the network exhibiting differential responses varies according to whether or not memory performance is enhanced or impaired. The collective findings we review indicate that multisensory associations formed via single-trial learning exert influences on later unisensory processing to promote distinct object representations that manifest as differentiable brain networks whose activity is correlated with memory performance. These influences occur incidentally, despite many intervening stimuli, and are distinguishable from the encoding/learning processes during the formation of the multisensory associations. The consequences of multisensory interactions that persist over time to impact memory retrieval and object discrimination.

That smells blue! Differences between colour associations for odours and odour-evocative words

Strong associations exist between specific odours and colours, and these associations have been found to be both consistent within populations and over time (Gilbertet al., 1996). Experimental manipulations of these associations have shown that both taste and odour perception rely heavily upon visual cues (e.g., Blackwell, 1995; Sakaiet al., 2005); participants often make errors in odour judgements when stimuli have been artificially coloured (Morrotet al., 2001), and the presence of a strongly-associated colour can greatly enhance the detection of an odour and the intensity of aromas or flavours (Zellner and Kautz, 1990; Zellner and Whitten, 1999), as well as preference and enjoyment (Herz, 2001; Herz and Beland, 2004). Such associations between colour and odour appear to be based on prior experience (Blackwell, 1995; Morrotet al., 2001; Sakaiet al., 2005; Stevenson and Oaten, 2008), and odours are usually perceived alongside visual, taste and tactile sensations, as well as higher order cues such as shape, size and object labelling. As such, an odour maybe perceived quite differently depending upon its current multisensory context, and experiencing an odour without these additional cues is likely to be different from experiencing the odour in a natural multisensory environment. Here we explore if odour-evocative words, rich in semantic connotations, differ in their colour associations compared to those associated with just odour. Twenty individuals were tested in each of four geographical locations; Germany, Malaysia, The Netherlands and the United States of America. Participants chose the three colours they most closely associated with both odours and odour-words from a chart of 36, using Xperiment software (www.xperiment.mobi). Preliminary results indicate that there were differences between odour-evocative words and odour cues in terms of the associated colours, for all populations.

Sound-Induced Enhancement of Low-Intensity Vision: Multisensory Influences on Human Sensory-Specific Cortices and Thalamic Bodies Relate to Perceptual Enhancement of Visual Detection Sensitivity

Combining information across modalities can affect sensory performance. We studied how co-occurring sounds modulate behavioral visual detection sensitivity (d'), and neural responses, for visual stimuli of higher or lower intensity. Co-occurrence of a sound enhanced human detection sensitivity for lower- but not higher-intensity visual targets. Functional magnetic resonance imaging (fMRI) linked this to boosts in activity-levels for sensory-specific visual and auditory cortex, plus multisensory superior temporal sulcus (STS), specifically for a lower-intensity visual event when paired with a sound. Thalamic structures in visual and auditory pathways, the lateral and medial geniculate bodies, respectively (LGB, MGB), showed a similar pattern. Subject-by-subject psychophysical benefits correlated with corresponding fMRI signals in visual, auditory, and multisensory regions. We also analyzed differential "coupling" patterns of LGB and MGB with other regions in the different experimental conditions. Effective-connectivity analyses showed enhanced coupling of sensory-specific thalamic bodies with the affected cortical sites during enhanced detection of lower-intensity visual events paired with sounds. Coupling strength between visual and auditory thalamus with cortical regions, including STS, covaried parametrically with the psychophysical benefit for this specific multisensory context. Our results indicate that multisensory enhancement of detection sensitivity for low-contrast visual stimuli by co-occurring sounds reflects a brain network involving not only established multisensory STS and sensory-specific cortex but also visual and auditory thalamus.

Disentangling unisensory and multisensory components in audiovisual integration using a novel multifrequency fMRI spectral analysis

Events in the world are mediated through multiple sensory inputs which are processed separately in specific unisensory areas according to the division-of-labor principle, and then need to be integrated to create a unified percept. How this is done is still not clear. For instance, recent evidence showed crossmodal activation in primary areas. We developed a novel approach to study multisensory integration using multifrequency spectral analysis to investigate the processing of audio and visual streams in a multisensory context. Auditory and visual stimuli were delivered in the same experimental condition, each in different presentation frequencies, and thus could be detected by applying Fourier spectral analysis in their different presentation frequencies. The cochleotopic and retinotopic organization of primary auditory and visual areas were found to remain intact in spite of the multisensory context. Auditory responses were also found in the Precuneus, suggesting that it might be a new auditory area responsive to pure tone stimuli, and serving as one end of a novel sensory preference gradient stretching across POG to the calcarine sulcus. Additional audiovisual areal convergence was detected both in areas in the middle of sensory preference gradients, and in primary auditory areas. Interestingly, the in/out synchronization rate of the auditory and visual streams yielded a third interaction frequency, which could be analyzed independently to reveal higher-order audiovisual interaction responses. These results were detected in one compact and natural multisensory experimental condition, which has several advantages over previous approaches. The method can be further implemented to study any type of interaction, within and across sensory modalities.

Enhanced Optic Flow Speed Discrimination While Walking: Contextual Tuning of Visual Coding

We tested the hypothesis that long-term adaptation to the normal contingencies between walking and its multisensory consequences (including optic flow) leads to enhanced discrimination of appropriate visual speeds during self-motion. In experiments 1 (task 1) and 2 a two-interval forced-choice procedure was used to compare the perceived speed of a simulated visual flow field viewed while walking with the perceived speed of a flow field viewed while standing. Both experiments demonstrated subtractive reductions in apparent speed. In experiments 1 and 3 discrimination thresholds were measured for optic flow speed while walking and while standing. Consistent with the optimal-coding hypothesis, speed discrimination for visual speeds near walking speed was enhanced during walking. Reduced sensitivity was found for slower visual speeds. The multisensory context of walking alters the coding of optic flow in a way that enhances speed discrimination in the expected range of flow speeds.

Integrative Studies: Teaching for the Twenty-first Century

THE INTEGRATIVE STUDIES MODEL for classroom activities that I propose and illustrate in this article combines the chronological focus of traditional disciplines with the thematic orientation of an interdisciplinary approach. A theme or pattern acts as the vehicle for organization. Students are presented with a variety of information about this theme, such as excerpts from government documents or newspapers, letters or bits of diaries, statistical information, poems, music, pictures, cartoons, works of art, common objects or archeological artifacts. The materials chosen should require different modes of perception in a rich multi-sensory context. Depending on the theme of the lesson, this information should not all come from one time or from one place. Some of it might even be current rather than from the past. The idea is to have the class develop an integrative paradigm that highlights the development over time of relationships between data. Rather than isolating specific bits of information and hoping that patterns of meaning will emerge, this model helps students recognize the interrelationships that shape their world. These patterns (themes) provide a context for understanding history and linking it relevantly to today. Thus, history will be remembered because students perceive it as meaningful and useful. Each mode of perception offers different insights into understanding the theme. Mathematics and the natural sciences offer literal, precise representa-