Amodal Information Research Articles

The cross-modal transfer of emotional information from voices to faces in 5-, 8- and 10-year-old children and adults: An eye-tracking study.

The present study examined the evolution of emotional cross-modal transfer throughout childhood compared to adulthood, using an experimental design first used with infants. We studied whether verbal children spontaneously look at emotional faces differently depending on the emotional voices previously heard, demonstrating a real intrinsic understanding of the emotion. Thus, sequences of emotional (happy vs. angry) cross-modal transfer were individually presented to 5-, 8- and 10-year-old children and adults. Spontaneous ocular behaviors toward the visual stimuli were recorded by eye-tracking. Results of the emotional cross-modal transfer suggested that participants looked spontaneously longer at the congruent face. However, this result was significantly revealed only as of age 8 with the happy voice and as of age 10 with the angry voice. Thus, the modulation of behavior indicators related to the control of the ability to extract amodal emotional information and spontaneously match the congruent information seems to increase with age and depends on the specific emotion presented. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Recognition of intersensory relationships based on duration in infants four, seven and 10 months old (Reconocimiento de relaciones intersensoriales basadas en la duración en bebés de cuatro, siete y 10 meses)

ABSTRACT Since birth, infants develop the ability to perceive a wide range of intersensory relations among various kinds of amodal temporal information. This study addresses the development of the ability to perceive duration-based intersensory relations. Three groups of infants, four, seven and 10 months old, participated in two trials of an intersensory 5preference task that used two audiovisual stimuli showing a woman executing two performances — with characteristics similar to the way adults interact with infants — consisting of sequences of sound-movement/silence-stillness of varying durations. The results show that infants at all three ages recognize the duration-based intersensory relation; however, recognition is expressed in different ways at each age according to the trial and the number of eye fixations recorded. These results are discussed in terms of the development of the ability to recognize duration-based intersensory relations and its relationship to between-subjects development.

Intersensory redundancy promotes infant detection of prosody in infant-directed speech

Prosody, or the intonation contours of speech, conveys emotion and intention to the listener and provides infants with an early basis for detecting meaning in speech. Infant-directed speech (IDS) is characterized by exaggerated prosody, slower tempo, and elongated pauses, all amodal properties detectable across the face and voice. Although speech is an audiovisual event, it has been studied primarily as a unimodal auditory stream without the synchronized dynamic face of the speaker. According to the intersensory redundancy hypothesis, redundancy across the senses facilitates perceptual learning of amodal information, including prosody. We predicted that young infants who are still learning to discriminate and categorize prosodic information would detect prosodic changes better in the presence of intersensory redundancy (i.e., synchronous audiovisual speech) than in its absence (i.e., unimodal auditory or asynchronous audiovisual speech). To test this hypothesis, 72 4-month-old infants were habituated to recordings of women reciting passages in IDS with prosody conveying either approval or prohibition and then were tested with recordings of a novel passage with either a change or no change in prosody. Infants who received bimodal synchronous stimulation exhibited significant visual recovery to the novel passage with a change in prosody, but not to a novel passage with no change in prosody. Infants in the unimodal auditory and bimodal asynchronous conditions did not exhibit visual recovery in either condition. Results support the hypothesis that intersensory redundancy facilitates detection and abstraction of invariant prosody across changes in linguistic content and likely serves as an early foundation for the detection of meaning in fluent speech.

Are 6-month-old human infants able to transfer emotional information (happy or angry) from voices to faces? An eye-tracking study.

The present study examined whether 6-month-old infants could transfer amodal information (i.e. independently of sensory modalities) from emotional voices to emotional faces. Thus, sequences of successive emotional stimuli (voice or face from one sensory modality -auditory- to another sensory modality -visual-), corresponding to a cross-modal transfer, were displayed to 24 infants. Each sequence presented an emotional (angry or happy) or neutral voice, uniquely, followed by the simultaneous presentation of two static emotional faces (angry or happy, congruous or incongruous with the emotional voice). Eye movements in response to the visual stimuli were recorded with an eye-tracker. First, results suggested no difference in infants’ looking time to happy or angry face after listening to the neutral voice or the angry voice. Nevertheless, after listening to the happy voice, infants looked longer at the incongruent angry face (the mouth area in particular) than the congruent happy face. These results revealed that a cross-modal transfer (from auditory to visual modalities) is possible for 6-month-old infants only after the presentation of a happy voice, suggesting that they recognize this emotion amodally.

Multisensory Motion Perception in 3-4 Month-Old Infants.

Human infants begin very early in life to take advantage of multisensory information by extracting the invariant amodal information that is conveyed redundantly by multiple senses. Here we addressed the question as to whether infants can bind multisensory moving stimuli, and whether this occurs even if the motion produced by the stimuli is only illusory. Three- to 4-month-old infants were presented with two bimodal pairings: visuo-tactile and audio-visual. Visuo-tactile pairings consisted of apparently vertically moving bars (the Barber Pole illusion) moving in either the same or opposite direction with a concurrent tactile stimulus consisting of strokes given on the infant’s back. Audio-visual pairings consisted of the Barber Pole illusion in its visual and auditory version, the latter giving the impression of a continuous rising or ascending pitch. We found that infants were able to discriminate congruently (same direction) vs. incongruently moving (opposite direction) pairs irrespective of modality (Experiment 1). Importantly, we also found that congruently moving visuo-tactile and audio-visual stimuli were preferred over incongruently moving bimodal stimuli (Experiment 2). Our findings suggest that very young infants are able to extract motion as amodal component and use it to match stimuli that only apparently move in the same direction.

Can audio-haptic speech be used to train better auditory speech perception?

Training with audio-visual speech improves subsequent auditory speech perception more so than training with auditory-alone speech (e.g., Bernstein, et al., 2013). What is the source of this bimodal training advantage? One explanation is that perceivers rely on learned bimodal associations. Alternatively, perceivers could be exploiting natural, amodal regularities available in both the auditory and visual signals. To test this question, multisensory training stimuli were tested for which observers had no bimodal associative experience. It is known that felt articulations —acquired by placing a hand on a speaker’s face— can provide information for speech perception (see Trielle et al., 2014, for a review). Importantly, these effects are found in participants with no prior experience perceiving speech through touch. If training with audio-haptic speech improves auditory speech perception, then this bimodal advantage cannot be due to learned associations but likely reflects sensitivity to amodal information. To test this hypothesis, participants either heard, or heard and felt, a speaker’s speech. Participants subsequently identified words from a set of novel audio-alone sentences. Preliminary data indicate that audio-haptic speech training improves subsequent auditory-only perception more than audio-only training. These results challenge a learned association account for the bimodal training advantage.

Auditory and motion metaphors have different scalp distributions: an ERP study.

While many links have been established between sensory-motor words used literally (kick the ball) and sensory-motor regions of the brain, it is less clear whether metaphorically used words (kick the habit) also show such signs of “embodiment.” Additionally, not much is known about the timing or nature of the connection between language and sensory-motor neural processing. We used stimuli divided into three figurativeness conditions—literal, metaphor, and anomalous—and two modality conditions—auditory (Her limousine was a privileged snort) and motion (The editorial was a brass-knuckle punch). The conditions were matched on a large number of potentially confounding factors including cloze probability. The electroencephalographic response to the final word of each sentence was measured at 64 electrode sites on the scalp of 22 participants and event-related potentials (ERPs) calculated. Analysis revealed greater amplitudes for metaphorical than literal sentences in both 350–500 ms and 500–650 ms timeframes. Results supported the possibility of different neural substrates for motion and auditory sentences. Greater differences for motion sentences were seen in the left posterior and left central electrode sites than elsewhere on the scalp. These findings are consistent with a sensory-motor neural categorization of language and with the integration of modal and amodal information during the N400 and P600 timeframes.

Book Review: Blind Vision: The Neuroscience of Visual Impairment

Blind Vision: The Neuroscience of Visual Impairment. Zaira Cattaneo and Tomaso Vecchi. Cambridge, MA: MIT Press, 2011, 269 pp., illustrated, $36. This book's title phrase, Blind Vision, is a concise summary of the book's contents, which comprise arguments and evidence that people who cannot see have the same kinds of perceptions, mental imagery, and knowledge as people who can see. Blindness is a big nuisance, but in most areas of everyday life it doesn't much change what people can notice, what people can remember, or what people can do. Authors Cattaneo and Vecchi take on the enormous task of synthesizing 50 years of cognitive psychology and cognitive neuroscience research conducted people who are blind. Like the best cognitive psychology, the work examines the cognitive processes and memory representations that underlie the skilled performance of functionally important tasks in everyday life. Like the best neuroscience, the work synthesizes evidence-based findings on the major brain imaging (fMRI and EEG, for example) and brain assessment (TMS, for example) methods that are used to localize brain regions that mediate perception and cognition, and it summarizes what happens to those brain regions when vision is absent or impaired. Cattaneo and Vecchi provide a comprehensive and balanced view of the cognitive neuroscience of blindness, acknowledging its current limitations and highlighting its potential applications. In doing so, they set the stage for others to apply the lessons gained to help learners learn and teachers teach. In terms of topical content, Blind Vision is about how perception, imagery, knowledge, working memory, and spatial cognition are used to accomplish the tasks of daily life that involve locomotion, wayfinding, and object identification and manipulation; as well as academic tasks including those involved in reading, science, and math. Wherever possible, the authors compare total blindness, low vision, and typical vision, and they identify perceptual and cognitive skills (tactile picture identification, for example) that appear to vary based on the age at onset of blindness and those (tactile discrimination, for example) that do not. And although it is not a major part of the book, the authors make an important contribution to the literature by synthesizing what is known about monocular vision-which, in our view, is an important category of low vision that is often overlooked by researchers and teachers alike. ORGANIZED BY THEME Cattaneo and Vecchi organize their topics around four major themes: Amodal information, sensory compensation and substitution, spatial imagery (not visual imagery), and brain plasticity. These themes form the core of the book and are mentioned in most of the chapters. We elaborate on each below. Amodal information Amodal information (meaning without modality) specifies properties of objects and events that are common in multiple senses. Temporal properties of events like rhythm, tempo, duration, and synchrony can be perceived by listening, looking, and touching. Consider, for example, that the cadence of a person's gait can be heard as well as seen by others; the viscosity of a liquid can be heard when it is poured and felt when it is stirred; and whether a smooth plate is metal or wood can be felt when it is touched and heard when it is struck a mallet. Different modalities of information specify the same properties, but they are not completely redundant and vary in their precision, breadth of field, and speed. Cattaneo and Vecchi point to the evidence that when amodal information is available, people pragmatically select the mode that is easy, in the sense that it is best matched to a given task. For example, pedestrians tend to rely on vision for guidance in street crossings, because with a single gaze we can simultaneously embrace an enormous amount of information, and our foveal acuity allows us to focus on very detailed characteristics of what we are perceiving (p. …

Mismatching amodal redundancy inhibits operant learning in 5-month-old infants

The current study examined the functional role redundant amodal information plays in an operant learning task in 5-month-old human infants. Prior studies have suggested that both simple and complex learning processes (discrimination, associative conditioning) are facilitated when amodal information is presented redundantly across sensory modalities. These studies, however, did not test whether the amodal information had to be similar across modalities for facilitation to occur. The current study examined how both matching and mismatching redundant amodal information about the shape of an object would influence learning of an operant response in human infants. Infants learned an operant kick response to move a mobile of cylinders while either holding a cylinder, a rectangular cube, or no object. Kick rate served as the dependent measure. The results showed that infants given mismatching redundant amodal information (e.g., viewed cylinders while holding a rectangular cube) showed inhibited operant learning. These results extend the Intersensory Redundancy Hypothesis by demonstrating that amodal redundancy can function in some instances to inhibit complex learning processes.

Redundant amodal properties facilitate operant learning in 3-month-old infants

The current study examined the role redundant amodal properties play in an operant learning task in 3-month-old human infants. Prior studies have suggested that the presence of redundant amodal information facilitates detection and discrimination of amodal properties and potentially functions to influence general learning processes such as associative conditioning. The current study examined how human infants use redundant amodal information (visual and haptic) about the shape of an object to influence learning of an operant response. Infants learned to kick to move a mobile of cylinders while either holding a cylinder, a rectangular cube, or no object. Kick rate served as the dependent measure. The results showed that infants given matching redundant amodal properties (e.g., viewed cylinders while holding a cylinder) showed facilitated operant learning whereas infants given mismatching redundant amodal properties showed inhibited operant learning. These results support and extend the Intersensory Redundancy Hypothesis by demonstrating that amodal redundancy influences complex learning processes such as operant conditioning.

Reducing expectations: Is an extension of current boundary extension theories needed?

Boundary extension is a phenomenon in which people remember seeing more of a recently-viewed image than was actually shown. Typically, close-angle images produce greater amounts of extension than wide-angle images, however, images must be perceived as having a continuous, yet truncated, view of the world. Current theories of boundary extension suggest the phenomenon involves a source monitoring error between mental representations created by sensory information present in the image and representations containing amodal information based on perceptual expectations. If this is the case images/scenes with reduced expectations to amodally continue should not produce or minimally produce boundary extension. In the current study we tested this idea by removing a number of factors thought to contribute to amodal continuation including familiarity, texture gradients, and occlusion information. We did this by using 2-D abstract shapes on random-dot backgrounds where the backgrounds were either occluded by the border or not and where the objects occluded the random-dots or not. We found boundary extension occurred in all conditions created to minimize amodal continuation. These results suggest modification of current boundary extension theories may be necessary to incorporate the present findings.

Speech Perception as a Multimodal Phenomenon

Speech perception is inherently multimodal. Visual speech (lip-reading) information is used by all perceivers and readily integrates with auditory speech. Imaging research suggests that the brain treats auditory and visual speech similarly. These findings have led some researchers to consider that speech perception works by extracting amodal information that takes the same form across modalities. From this perspective, speech integration is a property of the input information itself. Amodal speech information could explain the reported automaticity, immediacy, and completeness of audiovisual speech integration. However, recent findings suggest that speech integration can be influenced by higher cognitive properties such as lexical status and semantic context. Proponents of amodal accounts will need to explain these results.

Estimação de tempo em ratos submetidos a um procedimento de discriminação condicional

Num procedimento de discriminação condicional (DC) de característica positiva, um mesmo estímulo alvo A não é seguido de reforço quando apresentado sozinho, mas é seguido de reforço quando precedido de um estímulo X. Na DC de característica negativa, um mesmo estímulo alvo B é seguido de reforço quando apresentado sozinho, mas nunca quando precedido do estímulo Y. Nesta pesquisa, procurou-se examinar se uma informação amodal relativa a atributos básicos de estímulos, como a duração, tem capacidade de adquirir propriedades condicionais em animais. Ratos foram submetidos a procedimentos de DC positiva e negativa com estímulos X e Y de mesma modalidade física, uma luz, mas com durações diferentes: X= 15-seg e Y= 5-seg; as durações dos estímulos alvo foram A= 5-seg e B= 15-seg. Os resultados mostraram que os animais discriminaram as propriedades condicionais excitatórias (X) e inibitórias (Y) relativas, respectivamente, a duas durações diferentes de um mesmo estímulo, sustentando a hipótese de que a duração, um componente amodal de estímulo, pode ser associada a propriedades condicionais, em ratos.

The Development of Visual-Tactual Perception of Objects: Amodal Relations Provide the Basis for Learning Arbitrary Relations.

Infants of 4 and 6 months were familiarized with an object visually and tactually and then tested for visual-tactual matching on the basis of amodal and arbitrary relations or arbitrary relations only (Experiment 1). Results indicated that both the 4- and the 6-month-olds were capable of matching when amodal information specifying the common shape of the visual and tactual stimulation was available during test. In contrast, only the 6-month-olds were capable of matching on the basis of arbitrary relations between the haptically experienced shape of the object and its color or pattern. A control study (Experiment 2) demonstrated that the 4-month-olds were, in fact, able to discriminate the colors or patterns used. Results were interpreted as consistent with findings in the area of audiovisual event perception, demonstrating a developmental lag between detection of amodal and arbitrary relations. Finally, Experiment 3 explored the basis for 6-month-olds' detection of the arbitrary relations in Experiment 1....

Infants' response to the audible and visible properties of the human face. I: Role of lexical-syntactic content, temporal synchrony, gender, and manner of speech.

Four-, 6-, and 8-month-old infants’ perception of the multimodal features of the human face was investigated. First, infants were habituated to a visible and audible face of a person reciting a prepared script. Then they were tested by changing various features of just the audible, just the visible, or both components of the face. When features were changed, such as the lexical-syntactic content, the speaker’s gender, or the synchrony relation between the audible and visible components, the infants discriminated their multimodal and visible representation but not the audible one. When the manner of speech was changed from adult- to infant-directed, the 2 older groups discriminated all 3 types of changes but the 4-month-old infants only discriminated its visible and multimodal representation. Results show that speech-related exaggerated prosody cues facilitate detection of the audible features of multimodally represented faces but not until 6 months of age. The perceptual world of the infant is multimodal in nature. Objects and events often can be seen, heard, felt, smelled, or tasted. The features that simultaneously specify objects and events in different sensory modalities may b e amodal (i.e., equivalent across the different sensory modalities) or modalityspecific. For example, the human face can be specified by a host of amodal features (i.e., tempo, duration, and synchrony) because of correspondences created by the way the mouth moves while it produces sounds. It also can be specified concurrently by a host of modality-specific features such as size and color of eyebrows, hair color, lip size, overall configuration of the various facial features, skin color, and pitch and timbre of the voice. How infants respond to these kinds of amodal and modalityspecific features of multimodally specified objects and events has become a question of considerable research interest ( Lewkowicz & Lickliter, 1994). Most of the research designed to answer this question has focused on infants’ ability to integrate the multimodal attributes of events and objects on the basis o f amodal information. As a whole, this research has shown that many basic intersensory integrative abilities emerge during the first year of life (Lewkowicz & Lickliter, 1994). In a typical study o f intersensory integration, infants are presented with two, side-by-side, dynamic visual displays and a sound that corresponds to one of

Role of an abstract order schema in conceptual judgment.

Ss performed same-different judgments for which order information was logically irrelevant. In Experiments 1 and 2, the stimuli for the judgment task were derived from an ordered set of concepts from long-term memory (U.S. presidents ordered on the dimension of historical time); in Experiments 3 and 4 the stimuli were derived from two ordered sets of concepts. In the stimulus set for each experiment, there were several associate phrases for each concept and the task was to judge whether the phrases were paired with the same concept (Experiments 1, 2, and 3) or the same ordering (Experiment 4). The time to respond "different" decreased with the ordinal distance between the concepts even when the concepts belonged to different orderings. It is concluded that same-different judgments are based in part on amodal order information (i.e., not tied to any particular dimension). Two models of the implicated order schema are proposed and tested.