Amodal Processing Research Articles

Examining modal and amodal language processing in proficient bilinguals: evidence from the modality-switch paradigm.

Recent discussions have emphasized the significance of embodied processing in language comprehension. Nevertheless, continuous debates persist regarding the relative contribution of modal (embodied) and amodal (abstract) processing of language. The current study investigated the contribution of modal processing in the first (L1) and second (L2) language, hypothesizing higher level of abstract amodal symbol processing in L2 than L1, since the correspondence of L1 and L2 (i.e., the symbol-to-symbol assignment) is in the foreground when learning L2. We employed the modality-switch paradigm (Pecher et al., 2003) in both German and French versions with proficient sequential German and French bilinguals (N = 79). Participants were presented with noun-adjective pairs (e.g., keys - jingling) in both languages and decided whether the adjective could be applied to the noun. This task repeatedly requires switching modality between trials, (e.g., from auditory [keys - jingling] to olfactory [soap - perfumed]), typically causing switch costs on response latency as compared to maintaining the modality. Contrary to the hypothesis, we observed modality switch effects (MSE) in both L1 and L2. This result suggests that embodied language processing occurs not only in the first language but also extends to the second language thus challenging the assumption that L2 processing predominantly involves abstract amodal symbol processing. Notably, however, significant L1 and L2 MSEs were found for French, whereas for German already the L1 effect was rather weak (though significant); the corresponding L2 effect was not significant. Thus, the results hinted at differences between languages regarding the relative role of modal and amodal processing.

TMS Reveals Dynamic Interaction between Inferior Frontal Gyrus and Posterior Middle Temporal Gyrus in Gesture-Speech Semantic Integration.

Semantic processing is an amodal process with modality-specific information integrated in supramodal "convergence zones" or "semantic hub" with executive mechanisms that tailor semantic representation in a task-appropriate way. One unsolved question is how frontal control region dynamically interacts with temporal representation region in semantic integration. The present study addressed this issue by using inhibitory double-pulse transcranial magnetic stimulation over the left inferior frontal gyrus (IFG) or left posterior middle temporal gyrus (pMTG) in one of eight 40 ms time windows (TWs) (3 TWs before and 5 TWs after the identification point of speech), when human participants (12 females, 14 males) were presented with semantically congruent or incongruent gesture-speech pairs but merely identified the gender of speech. We found a TW-selective disruption of gesture-speech integration, indexed by the semantic congruency effect (i.e., a cost of reaction time because of semantic conflict), when stimulating the left pMTG in TW1, TW2, and TW7 but when stimulating the left IFG in TW3 and TW6. Based on the timing relationship, we hypothesize a two-stage gesture-speech integration circuit with a pMTG-to-IFG sequential involvement in the prelexical stage for activating gesture semantics and top-down constraining the phonological processing of speech. In the postlexical stage, an IFG-to-pMTG feedback signal might be implicated for the control of goal-directed representations and multimodal semantic unification. Our findings provide new insights into the dynamic brain network of multimodal semantic processing by causally revealing the temporal dynamics of frontal control and temporal representation regions.SIGNIFICANCE STATEMENT Previous research has identified differential functions of left inferior frontal gyrus (IFG) and posterior middle temporal gyrus (pMTG) in semantic control and semantic representation, respectively, and a causal contribution of both regions in gesture-speech integration. However, it remains largely unclear how the two regions dynamically interact in semantic processing. By using double-pulse transcranial magnetic stimulation to disrupt regional activity at specific time, this study for the first time revealed critical time windows when the two areas were causally involved in integrating gesture and speech semantics. Findings suggest a pMTG-IFG-pMTG neurocircuit loop in gesture-speech integration, which deepens current knowledge and inspires future investigation of the temporal dynamics and cognitive processes of the amodal semantic network.

The Anticipatory and Task-Driven Nature of Visual Perception.

Humans have a remarkable capacity to arrange and rearrange perceptual input according to different categorizations. This begs the question whether the categorization is exclusively a higher visual or amodal process, or whether categorization processes influence early visual areas as well. To investigate this we scanned healthy participants in a magnetic resonance imaging scanner during a conceptual decision task in which participants had to answer questions about upcoming images of animals. Early visual cortices (V1 and V2) contained information about the current visual input, about the granularity of the forthcoming categorical decision, as well as perceptual expectations about the upcoming visual stimulus. The middle temporal gyrus, the anterior temporal lobe, and the inferior frontal gyrus were also involved in the categorization process, constituting an attention and control network that modulates perceptual processing. These findings provide further evidence that early visual processes are driven by conceptual expectations and task demands.

Cross-Modal Transfer Following Auditory Task-Switching Training in Old Adults.

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory task-switching paradigm over four sessions (2880 total trials), to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks (active control) or not trained (passive control) allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level.

The Role of the Left and Right Anterior Temporal Poles in People Naming and Recognition

People’s identity recognition and the neural correlates underlying this process are still a matter of debate. While neuropsychological reports on single cases show a crucial role of the left anterior temporal lobe (ATL) in proper naming, and of the right ATL in people’s identification, reviews are less consistent. Moreover, it is still controversial whether familiarity and personal semantics access rely on amodal processes or follow modality-dependent paths. To disentangle these issues, we tested, in a parallel-group design, neurologically unimpaired subjects in two famous people recognition tasks after anodal tDCS over the left or right ATL or after a placebo stimulation condition. In the famous people recognition task, subjects were presented with visual (face recognition) or auditory (voice recognition) stimuli and subjects had to judge whether stimuli belonged to a famous or non-famous person (familiarity test); then, if the stimulus was recognized as famous, participants had to provide personal semantic information about the character; finally, to investigate proper naming, subjects were asked to name the famous person. While right ATL anodal tDCS increased accuracy in famous faces (but not voices) judgment and personal semantics retrieval, left ATL stimulation increased proper naming for both visual and auditory stimuli. Our data support a key role of the right ATL in famous people recognition and access to personal semantics from visual inputs, while the left ATL seems crucial for proper naming, which seems to occur at a later stage, when presentation modality no longer affects the process.

Cross-modal transfer after auditory task-switching training

Task-switching training was shown to improve performance not only for the trained tasks (i.e., reduced performance costs resulting from the task switches), but also for structurally similar (near transfer) or even dissimilar tasks (far transfer). However, it is still unclear whether the improvement is specific to the trained input modality or whether cognitive control occurs at an amodal processing level enabling transfer of set-shifting abilities to different input modalities. In this study, training and transfer was assessed for an auditory task-switching paradigm in which spoken words from different semantic categories were presented dichotically requiring participants to switch between two auditory categorization tasks. Cross-modal transfer of task-switching training was assessed in terms of the performance costs in a visual task-switching situation using tasks that were structurally similar to the trained tasks. The 4-day training significantly reduced the costs resulting from mixing the two auditory tasks, as compared to both an active (auditory single-task training) and a passive control group (no training). More importantly, the auditory task-switching training was also found to reduce the mixing costs for untrained visual tasks, indicating cross-modal transfer. This finding suggests that the improvement resulting from task-switching training is not specific to the trained stimulus modality, but it seems to be driven by a cognitive control mechanism operating at an amodal processing level. The training did not reveal any far-transfer effects to working memory, inhibition, or fluid intelligence, suggesting that the modality-independent enhancement of set-shifting does not generalize to other cognitive control functions.

Listening beyond seeing: Event-related potentials to audiovisual processing in visual narrative

Every day we integrate meaningful information coming from different sensory modalities, and previous work has debated whether conceptual knowledge is represented in modality-specific neural stores specialized for specific types of information, and/or in an amodal, shared system. In the current study, we investigated semantic processing through a cross-modal paradigm which asked whether auditory semantic processing could be modulated by the constraints of context built up across a meaningful visual narrative sequence. We recorded event-related brain potentials (ERPs) to auditory words and sounds associated to events in visual narratives—i.e., seeing images of someone spitting while hearing either a word (Spitting!) or a sound (the sound of spitting)—which were either semantically congruent or incongruent with the climactic visual event. Our results showed that both incongruent sounds and words evoked an N400 effect, however, the distribution of the N400 effect to words (centro-parietal) differed from that of sounds (frontal). In addition, words had an earlier latency N400 than sounds. Despite these differences, a sustained late frontal negativity followed the N400s and did not differ between modalities. These results support the idea that semantic memory balances a distributed cortical network accessible from multiple modalities, yet also engages amodal processing insensitive to specific modalities.

Age-Related Declines in Early Sensory Memory: Identification of Rapid Auditory and Visual Stimulus Sequences.

Age-related temporal-processing declines of rapidly presented sequences may involve contributions of sensory memory. This study investigated recall for rapidly presented auditory (vowel) and visual (letter) sequences presented at six different stimulus onset asynchronies (SOA) that spanned threshold SOAs for sequence identification. Younger, middle-aged, and older adults participated in all tasks. Results were investigated at both equivalent performance levels (i.e., SOA threshold) and at identical physical stimulus values (i.e., SOAs). For four-item sequences, results demonstrated best performance for the first and last items in the auditory sequences, but only the first item for visual sequences. For two-item sequences, adults identified the second vowel or letter significantly better than the first. Overall, when temporal-order performance was equated for each individual by testing at SOA thresholds, recall accuracy for each position across the age groups was highly similar. These results suggest that modality-specific processing declines of older adults primarily determine temporal-order performance for rapid sequences. However, there is some evidence for a second amodal processing decline in older adults related to early sensory memory for final items in a sequence. This selective deficit was observed particularly for longer sequence lengths and was not accounted for by temporal masking.

The role of the temporal poles in proper naming and people recognition: a tDCS study.

The role of the temporal poles in proper naming and people recognition: a tDCS study.

The origins of metamodality in visual object area LO: Bodily topographical biases and increased functional connectivity to S1

Recent evidence from blind participants suggests that visual areas are task-oriented and sensory modality input independent rather than sensory-specific to vision. Specifically, visual areas are thought to retain their functional selectivity when using non-visual inputs (touch or sound) even without having any visual experience. However, this theory is still controversial since it is not clear whether this also characterizes the sighted brain, and whether the reported results in the sighted reflect basic fundamental a-modal processes or are an epiphenomenon to a large extent. In the current study, we addressed these questions using a series of fMRI experiments aimed to explore visual cortex responses to passive touch on various body parts and the coupling between the parietal and visual cortices as manifested by functional connectivity. We show that passive touch robustly activated the object selective parts of the lateral–occipital (LO) cortex while deactivating almost all other occipital–retinotopic-areas. Furthermore, passive touch responses in the visual cortex were specific to hand and upper trunk stimulations. Psychophysiological interaction (PPI) analysis suggests that LO is functionally connected to the hand area in the primary somatosensory homunculus (S1), during hand and shoulder stimulations but not to any of the other body parts. We suggest that LO is a fundamental hub that serves as a node between visual-object selective areas and S1 hand representation, probably due to the critical evolutionary role of touch in object recognition and manipulation. These results might also point to a more general principle suggesting that recruitment or deactivation of the visual cortex by other sensory input depends on the ecological relevance of the information conveyed by this input to the task/computations carried out by each area or network. This is likely to rely on the unique and differential pattern of connectivity for each visual area with the rest of the brain.

Pay Attention!: Sluggish Multisensory Attentional Shifting as a Core Deficit in Developmental Dyslexia.

The aim of this review is to provide a background on the neurocognitive aspects of the reading process and review neuroscientific studies of individuals with developmental dyslexia, which provide evidence for amodal processing deficits. Hari, Renvall, and Tanskanen (2001) propose amodal sluggish attentional shifting (SAS) as a causal factor for temporal processing deficits in dyslexia. Undergirding this theory is the notion that when dyslexics are faced with rapid sequences of stimuli, their automatic attentional systems fail to disengage efficiently, which leads to difficulty when moving from one item to the next (Lallier et al., ). This results in atypical perception of rapid stimulus sequences. Until recently, the SAS theory, particularly the examination of amodal attentional deficits, was studied solely through the use of behavioural measures (Facoetti et al., ; Facoetti, Lorusso, Cattaneo, Galli, & Molteni, ). This paper examines evidence within the literature that provides a basis for further exploration of amodal SAS as an underlying deficit in developmental dyslexia.

Executive Functioning and Visual Working Memory

The matter of modality is controversially discussed in the context of working memory (WM). There is evidence that modality-specific processes are accompanied by amodal processes to some extent. We investigated the relationship between executive-functioning tasks and visual WM (VWM) and were especially interested in the issue of amodal processes. Our correlational analyses suggest modality-independent relations of the tasks. We also aimed to quantify to what extent executive functioning is meaningful for VWM performances. We therefore estimated the relationship between executive tasks and VWM performances in a healthy (n = 710) and a clinical traumatic brain injury sample (n = 151) as well as in the combined total sample. The results indicate a substantial relevance of the verbal task for VWM performances in the total and the clinical sample but a low relevance in the healthy sample. These results could support assumptions of resource-depending differences in the relations of executive functioning and VWM but need further validation due to limitations of our study.

Irrelevant Auditory and Visual Events Induce a Visual Attentional Blink

In the present study we investigated whether a task-irrelevant distractor can induce a visual attentional blink pattern. Participants were asked to detect only a visual target letter (A, B, or C) and to ignore the preceding auditory, visual, or audiovisual distractor. An attentional blink was observed regardless of the distractor modality. The magnitude of the attentional blink was greater when the target was preceded by a visual or an audiovisual distractor than when the target letter was preceded by an auditory distractor. The presence of a distractor-induced attentional blink regardless of the distractor modality suggests that the attentional blink phenomenon is at least partly due to an amodal processing limitation.

Resources required for processing ambiguous complex features in vision and audition are modality specific

Processing multiple complex features to create cohesive representations of objects is an essential aspect of both the visual and auditory systems. It is currently unclear whether these processes are entirely modality specific or whether there are amodal processes that contribute to complex object processing in both vision and audition. We investigated this using a dual-stream target detection task in which two concurrent streams of novel visual or auditory stimuli were presented. We manipulated the degree to which each stream taxed processing conjunctions of complex features. In two experiments, we found that concurrent visual tasks that both taxed conjunctive processing strongly interfered with each other but that concurrent auditory and visual tasks that both taxed conjunctive processing did not. These results suggest that resources for processing conjunctions of complex features within vision and audition are modality specific.

The Duration of a Co-Occurring Sound Modulates Visual Detection Performance in Humans

BackgroundThe duration of sounds can affect the perceived duration of co-occurring visual stimuli. However, it is unclear whether this is limited to amodal processes of duration perception or affects other non-temporal qualities of visual perception.Methodology/Principal FindingsHere, we tested the hypothesis that visual sensitivity - rather than only the perceived duration of visual stimuli - can be affected by the duration of co-occurring sounds. We found that visual detection sensitivity (d’) for unimodal stimuli was higher for stimuli of longer duration. Crucially, in a cross-modal condition, we replicated previous unimodal findings, observing that visual sensitivity was shaped by the duration of co-occurring sounds. When short visual stimuli (∼24 ms) were accompanied by sounds of matching duration, visual sensitivity was decreased relative to the unimodal visual condition. However, when the same visual stimuli were accompanied by longer auditory stimuli (∼60–96 ms), visual sensitivity was increased relative to the performance for ∼24 ms auditory stimuli. Across participants, this sensitivity enhancement was observed within a critical time window of ∼60–96 ms. Moreover, the amplitude of this effect correlated with visual sensitivity enhancement found for longer lasting visual stimuli across participants.Conclusions/SignificanceOur findings show that the duration of co-occurring sounds affects visual perception; it changes visual sensitivity in a similar way as altering the (actual) duration of the visual stimuli does.

Conceptual representations of perceptual knowledge

Many neuroimaging studies of semantic memory have argued that knowledge of an object's perceptual properties are represented in a modality-specific manner. These studies often base their argument on finding activation in the left-hemisphere fusiform gyrus—a region assumed to be involved in perceptual processing—when the participant is verifying verbal statements about objects and properties. In this paper, we report an extension of one of these influential papers—Kan, Barsalou, Solomon, Minor, and Thompson-Schill (2003)—and present evidence for an amodal component in the representation and processing of perceptual knowledge. Participants were required to verify object-property statements (e.g., “cat-whiskers?”; “bear-wings?”) while they were being scanned by functional magnetic resonance imaging (fMRI). We replicated Kan et al.'s activation in the left fusiform gyrus, but also found activation in regions of left inferior frontal gyrus (IFG) and middle-temporal gyrus, areas known to reflect amodal processes or representations. Further, only activations in the left IFG, an amodal area, were correlated with measures of behavioural performance.

Multiple Mechanisms for Temporal Processing

Multiple Mechanisms for Temporal Processing

Priming words with pictures: Neural correlates of semantic associations in a cross‐modal priming task using fMRI

In our everyday life we process information from different modalities simultaneously with great ease. With the current study we had the following goals: to detect the neural correlates of (1) automatic semantic processing of associates and (2) to investigate the influence of different visual modalities on semantic processing. Stimuli were presented with a short SOA (350 ms) as subjects performed a lexical decision task. To minimize the variance and increase homogeneity within our sample, only male subjects were measured. Three experimental conditions were compared while brain activation was measured with a 3 T fMRI scanner: related word-pairs (e.g., frame-picture), unrelated word-pairs (e.g., frame-car) as well as word-nonword pairs (e.g., frame-fubber). They were presented uni- (word-word) and cross-modally (picture-word). Behavioral data revealed a priming effect for cross-modal and unimodal word-pairs. On a neural level, the unimodal condition revealed response suppression in bilateral fronto-parietal regions. Cross-modal priming led to response suppression within the right inferior frontal gyrus. Common areas of deactivation for both modalities were found in bilateral fronto-tempo-parietal regions. These results suggest that the processing of semantic associations presented in different modalities lead to modality-specific activation caused by early access routes. However, common activation for both modalities refers to a common neural network for semantic processing suggesting amodal processing.

Simultaneous completions of modal and amodal figures: Visual evoked potentials reveal asymmetrical interference effects

Modal and amodal completion processes are thought to affect the emergence and the potency of each other. To see whether one dominates over the other, we measured the Npd, that is, the negativity of visual-evoked potentials whose amplitude increases with perceptual difficulty. In the first experiment, the Npds to illusory (modal) squares and diamonds placed over amodal diamonds and squares, respectively, were found to be greater when targets were the amodal figures than when targets were the modal figures, thus suggesting that modal figures created more interference. A second experiment showed that greater Npds were specific to interference and not to the greater difficulty at focusing on amodal figures rather than on modal figures. A third experiment showed that the interference also occurs with real figures in replacement of modal ones. Overall, the results suggest a certain dominance of modal completion over amodal completion when both occur in the same display.

感動詞「エー」を表出した表情と音声に対するマルチモーダルな感情認知

Human communication of emotions is achieved through both facial and vocal information. The purpose of this study is to investigate the dominant sensory modality in recognition of emotions to the multi-modal expression. In Experiment 1, expressions of happiness, surprise, sadness, or aversion was presented vocally, facially, or in both modalities through the expression of an interjectory word “eh”. Participants were required to judge the emotion that was expressed. In Experiment 2, recognition of conflicted emotions between modalities was investigated by combining different emotions between facial and vocal expression. Results of the two experiments indicated that the observers predominantly recognized happiness and surprise that was expressed facially rather than vocally. Furthermore, the expression of happiness was often mistaken as surprise, and the expression of sadness was often mistaken as aversion. Importantly, however, the reverse of these mistakes was little observed. Such the asymmetries of confusion were consistently obtained in every modality including bimodal presentations. This evidence is suggestive that an amodal processing system exists in multi-modal recognition of emotions.