Cross-modal Selective Attention Research Articles

The Neural Correlates of Visual and Auditory Cross-Modal Selective Attention in Aging.

Age-related deficits in selective attention have been demonstrated to depend on the sensory modality through which targets and distractors are presented. Some of these investigations suggest a specific impairment of cross-modal auditory selective attention. For the first time, this study is taking on a whole brain approach while including a passive perception baseline, to investigate the neural underpinnings of selective attention across age groups, and taking the sensory modality of relevant and irrelevant (i.e., distracting) stimuli into account. Sixteen younger (mean age = 23.3 years) and 14 older (mean age = 65.3 years), healthy participants performed a series of delayed match-to-sample tasks, in which participants had to selectively attend to visual stimuli, selectively attend to auditory stimuli, or passively view and hear both types of stimuli, while undergoing 3T fMRI. The imaging analyses showed that areas recruited by cross-modal visual and auditory selective attention in both age groups included parts of the dorsal attention and frontoparietal control networks (i.e., intraparietal sulcus, insula, fusiform gyrus, anterior cingulate, and inferior frontal cortex). Most importantly, activation throughout the brain did not differ across age groups, suggesting intact brain function during cross-modal selective attention in older adults. Moreover, stronger brain activation during cross-modal visual vs. cross-modal auditory selective attention was found in both age groups, which is consistent with earlier accounts of visual dominance. In conclusion, these results do not support the hypothesized age-related deficit of cross-modal auditory selective attention. Instead, they suggest that the underlying neural correlates of cross-modal selective attention are similar in younger and older adults.

Changes of Effective Connectivity in the Alpha Band Characterize Differential Processing of Audiovisual Information in Cross-Modal Selective Attention.

Cross-modal selective attention enhances the processing of sensory inputs that are most relevant to the task at hand. Such differential processing could be mediated by a swift network reconfiguration on the macroscopic level, but this remains a poorly understood process. To tackle this issue, we used a behavioral paradigm to introduce a shift of selective attention between the visual and auditory domains, and recorded scalp electroencephalographic signals from eight healthy participants. The changes in effective connectivity caused by the cross-modal attentional shift were delineated by analyzing spectral Granger Causality (GC), a metric of frequency-specific effective connectivity. Using data-driven methods of pattern-classification and feature-analysis, we found that a change in the α band (12Hz-15 Hz) of GC is a stable feature across different individuals that can be used to decode the attentional shift. Specifically, auditory attention induces more pronounced information flow in the α band, especially from the parietal-occipital areas to the temporal-parietal areas, compared to the case of visual attention, reflecting a reconfiguration of interaction in the macroscopic brain network accompanying different processing. Our results support the role of α oscillation in organizing the information flow across spatially-separated brain areas and, thereby, mediating cross-modal selective attention.

What Can Computational Models Learn From Human Selective Attention? A Review From an Audiovisual Unimodal and Crossmodal Perspective.

Selective attention plays an essential role in information acquisition and utilization from the environment. In the past 50 years, research on selective attention has been a central topic in cognitive science. Compared with unimodal studies, crossmodal studies are more complex but necessary to solve real-world challenges in both human experiments and computational modeling. Although an increasing number of findings on crossmodal selective attention have shed light on humans' behavioral patterns and neural underpinnings, a much better understanding is still necessary to yield the same benefit for intelligent computational agents. This article reviews studies of selective attention in unimodal visual and auditory and crossmodal audiovisual setups from the multidisciplinary perspectives of psychology and cognitive neuroscience, and evaluates different ways to simulate analogous mechanisms in computational models and robotics. We discuss the gaps between these fields in this interdisciplinary review and provide insights about how to use psychological findings and theories in artificial intelligence from different perspectives.

Neural practice effect during cross-modal selective attention: Supra-modal and modality-specific effects

Practice and experiences gradually shape the central nervous system, from the synaptic level to large-scale neural networks. In natural multisensory environment, even when inundated by streams of information from multiple sensory modalities, our brain does not give equal weight to different modalities. Rather, visual information more frequently receives preferential processing and eventually dominates consciousness and behavior, i.e., visual dominance. It remains unknown, however, the supra-modal and modality-specific practice effect during cross-modal selective attention, and moreover whether the practice effect shows similar modality preferences as the visual dominance effect in the multisensory environment. To answer the above two questions, we adopted a cross-modal selective attention paradigm in conjunction with the hybrid fMRI design. Behaviorally, visual performance significantly improved while auditory performance remained constant with practice, indicating that visual attention more flexibly adapted behavior with practice than auditory attention. At the neural level, the practice effect was associated with decreasing neural activity in the frontoparietal executive network and increasing activity in the default mode network, which occurred independently of the modality attended, i.e., the supra-modal mechanisms. On the other hand, functional decoupling between the auditory and the visual system was observed with the progress of practice, which varied as a function of the modality attended. The auditory system was functionally decoupled with both the dorsal and ventral visual stream during auditory attention while was decoupled only with the ventral visual stream during visual attention. To efficiently suppress the irrelevant visual information with practice, auditory attention needs to additionally decouple the auditory system from the dorsal visual stream. The modality-specific mechanisms, together with the behavioral effect, thus support the visual dominance model in terms of the practice effect during cross-modal selective attention.

Effects of perceptual load and socially meaningful stimuli on crossmodal selective attention in Autism Spectrum Disorder and neurotypical samples.

The present study examined whether increasing visual perceptual load differentially affected both Socially Meaningful and Non-socially Meaningful auditory stimulus awareness in neurotypical (NT, n = 59) adults and Autism Spectrum Disorder (ASD, n = 57) adults. On a target trial, an unexpected critical auditory stimulus (CAS), either a Non-socially Meaningful ('beep' sound) or Socially Meaningful ('hi') stimulus, was played concurrently with the presentation of the visual task. Under conditions of low visual perceptual load both NT and ASD samples reliably noticed the CAS at similar rates (77-81%), whether the CAS was Socially Meaningful or Non-socially Meaningful. However, during high visual perceptual load NT and ASD participants reliably noticed the meaningful CAS (NT = 71%, ASD = 67%), but NT participants were unlikely to notice the Non-meaningful CAS (20%), whereas ASD participants reliably noticed it (80%), suggesting an inability to engage selective attention to ignore non-salient irrelevant distractor stimuli in ASD.

Examining the role of a medial olivocochlear reflex elicitor on the attentional modulation of cochlear function

Selective attention has been shown to modulate cortical and subcortical neural representations of sound in the auditory systems of humans and research animals. Neuroanatomy of the auditory system suggests that cortical activity is capable of modulating cochlear responses to sound via corticofugal projections to the medial olivocochlear (MOC) efferent system. Several human studies using otoacoustic emissions (OAEs) suggest that selective attention can modulate cochlear responses to sound, but results across studies typically show small and inconsistent effects. Recent work in our laboratory has demonstrated much larger effects of cross-modal selective attention on cochlear processing than previously reported, likely due to improved methods for measuring MOC effects. One unanswered question is whether selective attention can modulate cochlear function directly, or whether it only modulates stimulus-elicited MOC activity. Our current study compares OAE magnitudes measured while participants attend to auditor...

Selective Attention and Sensory Modality in Aging: Curses and Blessings.

The notion that selective attention is compromised in older adults as a result of impaired inhibitory control is well established. Yet it is primarily based on empirical findings covering the visual modality. Auditory and especially, cross-modal selective attention are remarkably underexposed in the literature on aging. In the past 5 years, we have attempted to fill these voids by investigating performance of younger and older adults on equivalent tasks covering all four combinations of visual or auditory target, and visual or auditory distractor information. In doing so, we have demonstrated that older adults are especially impaired in auditory selective attention with visual distraction. This pattern of results was not mirrored by the results from our psychophysiological studies, however, in which both enhancement of target processing and suppression of distractor processing appeared to be age equivalent. We currently conclude that: (1) age-related differences of selective attention are modality dependent; (2) age-related differences of selective attention are limited; and (3) it remains an open question whether modality-specific age differences in selective attention are due to impaired distractor inhibition, impaired target enhancement, or both. These conclusions put the longstanding inhibitory deficit hypothesis of aging in a new perspective.

Cross-modal perceptual load: the impact of modality and individual differences.

Visual distractor processing tends to be more pronounced when the perceptual load (PL) of a task is low compared to when it is high [perpetual load theory (PLT); Lavie in J Exp Psychol Hum Percept Perform 21(3):451-468, 1995]. While PLT is well established in the visual domain, application to cross-modal processing has produced mixed results, and the current study was designed in an attempt to improve previous methodologies. First, we assessed PLT using response competition, a typical metric from the uni-modal domain. Second, we looked at the impact of auditory load on visual distractors, and of visual load on auditory distractors, within the same individual. Third, we compared individual uni- and cross-modal selective attention abilities, by correlating performance with the visual Attentional Network Test (ANT). Fourth, we obtained a measure of the relative processing efficiency between vision and audition, to investigate whether processing ease influences the extent of distractor processing. Although distractor processing was evident during both attend auditory and attend visual conditions, we found that PL did not modulate processing of either visual or auditory distractors. We also found support for a correlation between the uni-modal (visual) ANT and our cross-modal task but only when the distractors were visual. Finally, although auditory processing was more impacted by visual distractors, our measure of processing efficiency only accounted for this asymmetry in the auditory high-load condition. The results are discussed with respect to the continued debate regarding the shared or separate nature of processing resources across modalities.

The role of crossmodal competition and dimensional overlap in crossmodal attention switching

Crossmodal selective attention was investigated in a cued task switching paradigm using bimodal visual and auditory stimulation. A cue indicated the imperative modality. Three levels of spatial S–R associations were established following perceptual (location), structural (numerical), and conceptual (verbal) set-level compatibility. In Experiment 1, participants switched attention between the auditory and visual modality either with a spatial-location or spatial-numerical stimulus set. In the spatial-location set, participants performed a localization judgment on left vs. right presented stimuli, whereas the spatial-numerical set required a magnitude judgment about a visually or auditorily presented number word. Single-modality blocks with unimodal stimuli were included as a control condition. In Experiment 2, the spatial-numerical stimulus set was replaced by a spatial-verbal stimulus set using direction words (e.g., “left”). RT data showed modality switch costs, which were asymmetric across modalities in the spatial-numerical and spatial-verbal stimulus set (i.e., larger for auditory than for visual stimuli), and congruency effects, which were asymmetric primarily in the spatial-location stimulus set (i.e., larger for auditory than for visual stimuli). This pattern of effects suggests task-dependent visual dominance.

Age-equivalent top-down modulation during cross-modal selective attention.

Selective attention involves top-down modulation of sensory cortical areas, such that responses to relevant information are enhanced whereas responses to irrelevant information are suppressed. Suppression of irrelevant information, unlike enhancement of relevant information, has been shown to be deficient in aging. Although these attentional mechanisms have been well characterized within the visual modality, little is known about these mechanisms when attention is selectively allocated across sensory modalities. The present EEG study addressed this issue by testing younger and older participants in three different tasks: Participants attended to the visual modality and ignored the auditory modality, attended to the auditory modality and ignored the visual modality, or passively perceived information presented through either modality. We found overall modulation of visual and auditory processing during cross-modal selective attention in both age groups. Top-down modulation of visual processing was observed as a trend toward enhancement of visual information in the setting of auditory distraction, but no significant suppression of visual distraction when auditory information was relevant. Top-down modulation of auditory processing, on the other hand, was observed as suppression of auditory distraction when visual stimuli were relevant, but no significant enhancement of auditory information in the setting of visual distraction. In addition, greater visual enhancement was associated with better recognition of relevant visual information, and greater auditory distractor suppression was associated with a better ability to ignore auditory distraction. There were no age differences in these effects, suggesting that when relevant and irrelevant information are presented through different sensory modalities, selective attention remains intact in older age.

Effects of cross-modal selective attention on the sensory periphery: Cochlear sensitivity is altered by selective attention

There is increasing evidence that alterations in the focus of attention result in changes in neural responding at the most peripheral levels of the auditory system. To date, however, those studies have not ruled out differences in task demands or overall arousal in explaining differences in responding across intermodal attentional conditions. The present study sought to compare changes in the response of cochlear outer hair cells, employing distortion product otoacoustic emissions (DPOAEs), under different, balanced conditions of intermodal attention. DPOAEs were measured while the participants counted infrequent, brief exemplars of the DPOAE primary tones (auditory attending), and while counting visual targets, which were instances of Gabor gradient phase shifts (visual attending). Corroborating an earlier study from our laboratory, the results show that DPOAEs recorded in the auditory-ignoring condition were significantly higher in overall amplitude, compared with DPOAEs recorded while participants attended to the eliciting primaries; a finding in apparent contradiction with more central measures of intermodal attention. Also consistent with our previous findings, DPOAE rapid adaptation, believed to be mediated by the medial olivocochlear efferents (MOC), was unaffected by changes in intermodal attention. The present findings indicate that manipulations in the conditions of attention, through the corticofugal pathway, and its last relay to cochlear outer hair cells (OHCs), the MOC, alter cochlear sensitivity to sound. These data also suggest that the MOC influence on OHC sensitivity is composed of two independent processes, one of which is under attentional control.

Now you see it, now you don't: Evidence for age-dependent and age-independent cross-modal distraction.

Age-related deficits in selective attention have often been demonstrated in the visual modality and, to a lesser extent, in the auditory modality. In contrast, a mounting body of evidence has suggested that cross-modal selective attention is intact in aging, especially in visual tasks that require ignoring the auditory modality. Our goal in this study was to investigate age-related differences in the ability to ignore cross-modal auditory and visual distraction and to assess the role of cognitive control demands thereby. In a set of two experiments, 30 young (mean age = 23.3 years) and 30 older adults (mean age = 67.7 years) performed a visual and an auditory n-back task (0 ≤ n ≤ 2), with and without cross-modal distraction. The results show an asymmetry in cross-modal distraction as a function of sensory modality and age: Whereas auditory distraction did not disrupt performance on the visual task in either age group, visual distraction disrupted performance on the auditory task in both age groups. Most important, however, visual distraction was disproportionately larger in older adults. These results suggest that age-related distraction is modality dependent, such that suppression of cross-modal auditory distraction is preserved and suppression of cross-modal visual distraction is impaired in aging.

Variations of response time in a selective attention task are linked to variations of functional connectivity in the attentional network

Although variations of response time (RT) within a particular experimental condition are typically ignored, they may sometimes reflect meaningful changes in the efficiency of cognitive and neural processes. In the present study, we investigated whether trial-by-trial variations of response time (RT) in a cross-modal selective attention task were associated with variations of functional connectivity between brain regions that are thought to underlie attention. Sixteen healthy young adults performed an audiovisual selective attention task, which involved attending to a relevant visual letter while ignoring an irrelevant auditory letter, as we recorded their brain activity using functional magnetic resonance imaging (fMRI). In line with predictions, variations of RT were associated with variations of functional connectivity between the anterior cingulate cortex and various other brain regions that are posited to underlie attentional control, such as the right dorsolateral prefrontal cortex and bilateral regions of the posterior parietal cortex. They were also linked to variations of functional connectivity between anatomically early and anatomically late regions of the relevant-modality visual cortex whose communication is thought to be modulated by attentional control processes. By revealing that variations of RT in a selective attention task are linked to variations of functional connectivity in the attentional network, the present findings suggest that variations of attention may contribute to trial-by-trial fluctuations of behavioral performance.

Cross-modal selective attention effects on steady-state visual evoked potentials (SSVEPs)

Cross-modal selective attention effects on steady-state visual evoked potentials (SSVEPs)

Crossmodal effects of Guqin and piano music on selective attention: An event-related potential study

To compare the effects of music from different cultural environments (Guqin: Chinese music; piano: Western music) on crossmodal selective attention, behavioral and event-related potential (ERP) data in a standard two-stimulus visual oddball task were recorded from Chinese subjects in three conditions: silence, Guqin music or piano music background. Visual task data were then compared with auditory task data collected previously. In contrast with the results of the auditory task, the early (N1) and late (P300) stages exhibited no differences between Guqin and piano backgrounds during the visual task. Taking our previous study and this study together, we can conclude that: although the cultural-familiar music influenced selective attention both in the early and late stages, these effects appeared only within a sensory modality (auditory) but not in cross-sensory modalities (visual). Thus, the musical cultural factor is more obvious in intramodal than in crossmodal selective attention.

Momentary reductions of attention permit greater processing of irrelevant stimuli

Momentary reductions of attention can have extremely adverse outcomes, but it remains unclear whether increased distraction from irrelevant stimuli contributes to such outcomes. To investigate this hypothesis, we examined trial-by-trial relationships between brain activity and response time in twenty healthy adults while they performed a cross-modal selective attention task. In each trial, participants identified a relevant visual letter while ignoring an irrelevant auditory letter, which was mapped either to the same response as the visual letter (congruent trials) or to a different response (incongruent trials). As predicted, reductions of attention (i.e., increases of response time) were associated not only with decreased activity in sensory regions that processed the relevant visual stimuli, suggesting a failure to enhance the processing of those stimuli, but also with increased activity in sensory regions that processed the irrelevant auditory stimuli, suggesting a failure to suppress the processing of those stimuli. Reductions of attention were also linked to larger increases of activity in incongruent than in congruent trials in anterior cingulate regions that detect response conflict, suggesting that failing to suppress the sensory processing of the irrelevant auditory stimuli during attentional reductions allowed those stimuli to more readily activate conflicting responses in incongruent trials. These findings indicate that heightened levels of distraction during momentary reductions of attention likely stem, at least in part, from increased processing of irrelevant stimuli.

Switching attention between modalities: further evidence for visual dominance

The present study examined cross-modal selective attention using a task-switching paradigm. In a series of experiments, we presented lateralized visual and auditory stimuli simultaneously and asked participants to make a spatial decision according to either the visual or the auditory stimulus. We observed consistent cross-modal interference in the form of a spatial congruence effect. This effect was asymmetrical, with higher costs when responding to auditory than to visual stimuli. Furthermore, we found stimulus-modality-shift costs, indicating a persisting attentional bias towards the attended stimulus modality. We discuss our findings with respect to visual dominance, directed-attention accounts, and the modality-appropriateness hypothesis.

Preservation of crossmodal selective attention in healthy aging

The goal of the present study was to determine if older adults benefited from attention to a specific sensory modality in a voluntary attention task and evidenced changes in voluntary or involuntary attention when compared to younger adults. Suppressing and enhancing effects of voluntary attention were assessed using two cued forced-choice tasks, one that asked participants to localize and one that asked them to categorize visual and auditory targets. Involuntary attention was assessed using the same tasks, but with no attentional cues. The effects of attention were evaluated using traditional comparisons of means and Cox proportional hazards models. All analyses showed that older adults benefited behaviorally from selective attention in both visual and auditory conditions, including robust suppressive effects of attention. Of note, the performance of the older adults was commensurate with that of younger adults in almost all analyses, suggesting that older adults can successfully engage crossmodal attention processes. Thus, age-related increases in distractibility across sensory modalities are likely due to mechanisms other than deficits in attentional processing.

Crossmodal attention effects on brain responses to different stimulus classes

BackgroundAttending to a point in space in one modality may facilitate processing to information from the same region in another modality. The involvement of sensory-specific cortical areas in intramodal and crossmodal selective spatial attention can be assessed with event-related brain potentials (ERPs).ERPs were recorded in two groups of young participants (each n = 11). Stimulus sequences comprised visual standard stimuli (p = 0.4, a white square), visual deviant stimuli (p = 0.05, a white square with small black rectangle) and visual novel stimuli (p = 0.05, different multicolored checkerboards) as well as auditory standard (p = 0.4, 800 Hz tone), deviant (p = 0.05, 900 Hz tone), and novel (p = 0.05, random combination of three sine-wave tones) stimuli, occurring in random order at locations 30 degrees left and right of a fixation point. The "auditory group" of participants attended either to the left or to the right speaker in order to respond to the infrequent auditory deviants at that location by a speeded button press. Visual stimuli were irrelevant for this group. The "visual group" had the analogue task for the visual modality. For these participants auditory stimuli were irrelevant throughout the experiment.ResultsERPs showed a typical enhancement of early sensory specific components by intramodal spatial attention (visual group: visual P1 and N1; auditory group: auditory Nd). Crossmodal spatial attention effects included a modulation of the Nd to auditory standards in the visual group and a modulation of the P1 to visual novels and N1 for visual standards for the auditory group. Similar to previous studies crossmodal spatial attention effects on visual standard and novel stimuli also included a frontocentral positivity in the 200–400 ms range that was not seen for intramodal spatial visual attention suggesting involvement of later supramodal areas.ConclusionThese findings are consistent with an action of crossmodal spatial attention on early, sensory specific processing stages.

Vision and touch in ageing: Crossmodal selective attention and visuotactile spatial interactions

We investigated whether ageing affects crossmodal selective attention (the ability to focus on a relevant sensory modality and ignore an irrelevant modality) and the spatial constraints on such selective processing. Three groups of 24 participants were tested: Young (19–25 years), Young-Old (65–72 years) and Old-Old (76–92 years). The participants had to judge the elevation of vibrotactile targets (upper/index finger and lower/thumb), presented randomly to either hand while ignoring concurrent visual distractors. In a second task, the role of the target and distractor modalities was reversed. Crossmodal selective attention was assessed by comparing performance in the presence versus absence of distractors. Spatial constraints on selective attention were also investigated by comparing the effect of distractors presented on the same versus opposite side as the target. When attending to touch, the addition of visual distractors had a significantly larger effect on error rates in both of the older groups as compared to the Young group. This indicates that ageing has a detrimental effect on crossmodal selective attention. In all three age groups, performance was impaired when the target and distractor were presented at incongruent as compared to congruent elevations in both tasks. This congruency effect was modulated by the relative spatial location of the target and distractor in certain conditions for the Young and the Young-Old group. That is, participants in the two younger age groups found it harder to attend selectively to targets in one modality, when distractor stimuli came from the same side rather than from the opposite side. However, no significant spatial modulation was found in the Old-Old group. This suggests that ageing may also compromise spatial aspects of crossmodal selective attention.