Modality-specific Processing Research Articles

Modality matters: Three auditory conflict tasks to measure individual differences in attention control.

Early work on selective attention used auditory-based tasks, such as dichotic listening, to shed light on capacity limitations and individual differences in these limitations. Today, there is great interest in individual differences in attentional abilities, but the field has shifted towards visual-modality tasks. Furthermore, most conflict-based tests of attention control lack reliability due to low signal-to-noise ratios and the use of difference scores. Critically, it is unclear to what extent attention control generalizes across sensory modalities, and without reliable auditory-based tests, an answer to this question will remain elusive. To this end, we developed three auditory-based tests of attention control that use an adaptive response deadline (DL) to account for speed-accuracy trade-offs: Auditory Simon DL, Auditory Flanker DL, and Auditory Stroop DL. In a large sample (N = 316), we investigated the psychometric properties of the three auditory conflict tasks, tested whether attention control is better modeled as a unitary factor or modality-specific factors, and estimated the extent to which unique variance in modality-specific factors contributed incrementally to the prediction of dichotic listening and multitasking performance. Our analyses indicated that the auditory conflict tasks have strong psychometric properties and demonstrate convergent validity with visual tests of attention control. Auditory and visual attention control factors were highly correlated (r = .81)-even after controlling for perceptual processing speed (r = .75). Modality-specific attention control factors accounted for unique variance in modality-matched criterion measures, but the majority of the explained variance was modality-general. The results suggest an interplay between modality-general attention control and modality-specific processing.

Mechanisms of long-term repetition priming in recognising speech in noise

ABSTRACT Recognition of speech in noise is facilitated when spoken sentences are repeated a few minutes later, but the levels of representation involved in this effect have not been specified. Three experiments tested whether the effect would transfer across modalities and languages. In Experiment 1, participants listened to sets of high- and low-constraint sentences and read other sets in an encoding phase. At test, these sentences and new sentences were presented in noise, and participants attempted to report the final word of each sentence. Recognition was more accurate for repeated than for new sentences in both modalities. Experiment 2 was identical except for the implementation of an articulatory suppression task at encoding to reduce phonological recoding during reading. The cross-modal repetition priming effect persisted but was weaker than when the modality was the same at encoding and test. Experiment 3 showed that the repetition priming effect did not transfer across languages in bilinguals. Taken together, the results indicate that the facilitated recognition of repeated speech is based on a combination of modality-specific processes at the phonological word form level and modality-general processes at the lemma level of lexical representation, but the semantic level of representation is not involved.

Contextual effects on duration perception are modality-specific.

The perception of magnitude, crucial for a mental representation of the physical world, is often subject to significant biases. Many of these biases are similar across sensory modalities, implying a generalized perception of magnitude. At the same time, some physical magnitudes might have a dedicated modality-specific calibration mechanism to enhance perceptual sensitivity. We examined this question of generalized versus modality-specific processes testing between- and within-modalities' contextual effects on the perception of magnitude. In a constant stimuli procedure, a central standard was embedded in shorter and longer contextual standards. These contextual standards were sampled in either a relatively wider or narrower range of durations. Participants were asked to determine which of the two consecutive durations was longer. Better perceptual sensitivity was found for narrower contexts, with stronger effects in trials in which the standard was presented first. Interestingly, narrower context enhanced sensitivity for standards within the same modality but had no effect on standards of another modality. A unidirectional transfer of contextual effects was observed under certain conditions from auditory, the dominant modality in performing temporal judgments, to vision. These results suggest that the perceptual system appears to develop modality-specific calibration mechanisms, most likely, to enhance perceptual sensitivity and maintain sensory specialization. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Early, emotional and embodied? Processing of emotional words and body words in the native and a second language – evidence from early event-related brain potential modulation and rapid serial visual presentation

ABSTRACT Visual processing of emotional words modulates early event-related potentials (ERPs) such as the early posterior negativity (EPN). Questions remain as to whether this modulation reflects modality-specific processing, preferentially elicited by emotional words of the native language (L1). This study investigates the modulation of early ERPs during rapid serial visual presentation (RSVP) of adjectives or nouns referring to emotional feeling states, neutral traits, to an overweight or lean body or to concrete body parts or neutral objects, presented in the L1 and the second language (L2). Word ratings in the L2 were assessed in a pilot study. The N100 and the EPN were modulated by the emotional valence of the stimuli irrespective of the word class or the task (silent reading vs. word counting). The results suggest that early affective appraisal is obligatory, not restricted to privileged categories of linguistic information (emotions) or solely found for the embodied language (L1).

The Appraisal Principle in Multimedia Learning: Impact of Appraisal Processes, Modality, and Codality

This paper presents two experiments examining the influences of media-specific appraisal and attribution on multimedia learning. The first experiment compares four different versions of learning material (text, text with images, animation with text, and animation with audio). Results reveal that the attributed type of appraisal, (i.e., the subjective impression of whether a medium is easy or difficult to learn with) impacts invested mental effort and learning outcomes. Though there was no evidence for the modality effect in the first experiment, we were able to identify it in a second study. We were also able to replicate appraisal and attribution findings from study 1 in study 2: if media appraisal leads to the result that learning with a specific medium is difficult, more mental effort will be invested in information processing. Consequently, learning outcomes are better, and learners are more likely to attribute knowledge acquisition to their own abilities. Outcomes also indicate that the modality effect can be explained by avoidance of split-attention rather than modality-specific information processing in working memory.

Eye-closure effects and the influence of short-term storage and processing capacity on episodic memory

ABSTRACT Two experiments investigated differences in short-term storage and processing capacity on the magnitude of eye-closure effects on episodic memory. Experiment 1 compared individuals with high (vs. low) forward and backward spans in the free-recall of words retrieved under both eyes closed and open conditions. Main effects of both forward and backward span capacity (greater recall for the high span group) and eye-closure (higher recall with eyes closed) were found. Eye-closure was also associated with more “remember” responses. Experiment 2 compared individuals with high (vs. low) reading spans and found both main effects for reading span and eye-closure (greater recall for the high span group and with eyes closed). Remember responses were associated with both high reading span and eye-closure. The absence of interactions is discussed in terms of explanations of eye-closure effects that differentiate between modality-general and modality-specific processes.

Resetting of Auditory and Visual Segregation Occurs After Transient Stimuli of the Same Modality.

In the presence of a continually changing sensory environment, maintaining stable but flexible awareness is paramount, and requires continual organization of information. Determining which stimulus features belong together, and which are separate is therefore one of the primary tasks of the sensory systems. Unknown is whether there is a global or sensory-specific mechanism that regulates the final perceptual outcome of this streaming process. To test the extent of modality independence in perceptual control, an auditory streaming experiment, and a visual moving-plaid experiment were performed. Both were designed to evoke alternating perception of an integrated or segregated percept. In both experiments, transient auditory and visual distractor stimuli were presented in separate blocks, such that the distractors did not overlap in frequency or space with the streaming or plaid stimuli, respectively, thus preventing peripheral interference. When a distractor was presented in the opposite modality as the bistable stimulus (visual distractors during auditory streaming or auditory distractors during visual streaming), the probability of percept switching was not significantly different than when no distractor was presented. Conversely, significant differences in switch probability were observed following within-modality distractors, but only when the pre-distractor percept was segregated. Due to the modality-specificity of the distractor-induced resetting, the results suggest that conscious perception is at least partially controlled by modality-specific processing. The fact that the distractors did not have peripheral overlap with the bistable stimuli indicates that the perceptual reset is due to interference at a locus in which stimuli of different frequencies and spatial locations are integrated.

How does inattention affect written and spoken language processing?

The classic cocktail party effect suggests that some, but probably not all levels of language processing can proceed without attention. We used whole-brain functional MRI to investigate how modality-specific and modality-independent language areas are modulated by the withdrawal of attention to another sensory modality (e.g., attending to vision during the presentation of auditory sentences, or vice-versa). We tested the hypotheses that inattention may abolish sentence-level integration and eliminate top-down effects. In both written and spoken modalities, language processing was strongly modulated by the distraction of attention, but this inattention effect varied considerably depending on the area and hierarchical level of language processing. Under inattention, a bottom-up activation remained in early modality-specific areas, particularly in superior temporal spoken-language areas, but the difference between sentences and words lists vanished. Under both attended and unattended conditions, ventral temporal cortices were activated in a top-down manner by spoken language more than by control stimuli, reaching posteriorily the Visual Word Form Area. We conclude that inattention prevents sentence-level syntactic and semantic integration, but preserves some top-down crossmodal processing, plus a large degree of bottom-up modality-specific processing, including a ventral occipito-temporal specialization for letter strings in a known alphabet.

Distractor-induced deafness: The effect of multiple auditory distractors on conscious target processing

Conscious access to a target stimulus embedded in a rapid serial visual presentation can be impaired by the preceding presentation of multiple task-irrelevant distractors. While this phenomenon – labeled distractor-induced blindness (DIB) – is established in vision, it is unknown whether a similar effect can be observed in the auditory modality. Considering the differences in the processing of visual and auditory stimuli, modality-specific effects in the inhibitory mechanisms triggered by distractors can be expected. First, we aimed to find evidence for a distractor-induced deafness (DID) for auditory targets in a behavioral experiment. The target was defined by a transient increase in amplitude in a continuous sinusoidal tone, which was to be detected if accompanied or preceded by a deviant tone (cue). Both events were embedded in separate streams in a binaural rapid serial auditory presentation. Distractors preceded the cue and shared the target's features. As previously observed for DIB, a failure to detect the auditory target critically relied on the presentation of multiple distractor episodes. This DID effect was followed up in a subsequent event-related brain potentials (ERP) study to identify the signature of target detection. In contrast to missed targets, hits were characterized by a larger frontal negativity and by a more pronounced centro-parietal P3b wave. Whereas the latter process was also observed in the visual domain, indicating a post-perceptual updating process, the frontal negativity was exclusively observed for auditory DID. This modality-specific process might signal that early attentional control processes support conscious access to relevant auditory events.

Minimal Overlap in Language Control Across Production And Comprehension: Evidence from Read-Aloud Versus Eye-Tracking Tasks.

Bilinguals are remarkable at language control-switching between languages only when they want. However, language control in production can involve switch costs. That is, switching to another language takes longer than staying in the same language. Moreover, bilinguals sometimes produce language intrusion errors, mistakenly producing words in an unintended language (e.g., Spanish-English bilinguals saying "pero" instead of "but"). Switch costs are also found in comprehension. For example, reading times are longer when bilinguals read sentences with language switches compared to sentences with no language switches. Given that both production and comprehension involve switch costs, some language-control mechanisms might be shared across modalities. To test this, we compared language switch costs found in eye-movement measures during silent sentence reading (comprehension) and intrusion errors produced when reading aloud switched words in mixed-language paragraphs (production). Bilinguals who made more intrusion errors during the read-aloud task did not show different switch cost patterns in most measures in the silent-reading task, except on skipping rates. We suggest that language switching is mostly controlled by separate, modality-specific processes in production and comprehension, although some points of overlap might indicate the role of domain general control and how it can influence individual differences in bilingual language control.

Modality-general and modality-specific processes in hallucinations.

There is a growing recognition in psychosis research of the importance of hallucinations in modalities other than the auditory. This has focused attention on cognitive and neural processes that might be shared by, and which might contribute distinctly to, hallucinations in different modalities. In this article, I address some issues around the modality-generality of cognitive and neural processes in hallucinations, including the role of perceptual and reality-monitoring systems, top-down and bottom-up processes in relation to the psychological and neural substrates of hallucinations, and the phenomenon of simultaneous multimodal hallucinations of the same entity. I suggest that a functional systems approach, inspired by some neglected aspects of the writings of A. R. Luria, can help us to understand patterns of hallucinatory experience across modalities and across clinical and non-clinical groups. Understanding the interplay between modality-general and modality-specific processes may bear fruit for improved diagnosis and therapeutic approaches to dealing with distressing hallucinations.

Interrelations Between Temporal and Spatial Cognition: The Role of Modality-Specific Processing.

Temporal and spatial representations are not independent of each other. Two conflicting theories provide alternative hypotheses concerning the specific interrelations between temporal and spatial representations. The asymmetry hypothesis (based on the conceptual metaphor theory, Lakoff and Johnson, 1980) predicts that temporal and spatial representations are asymmetrically interrelated such that spatial representations have a stronger impact on temporal representations than vice versa. In contrast, the symmetry hypothesis (based on a theory of magnitude, Walsh, 2003) predicts that temporal and spatial representations are symmetrically interrelated. Both theoretical approaches have received empirical support. From an embodied cognition perspective, we argue that taking sensorimotor processes into account may be a promising steppingstone to explain the contradictory findings. Notably, different modalities are differently sensitive to the processing of time and space. For instance, auditory information processing is more sensitive to temporal than spatial information, whereas visual information processing is more sensitive to spatial than temporal information. Consequently, we hypothesized that different sensorimotor tasks addressing different modalities may account for the contradictory findings. To test this, we critically reviewed relevant literature to examine which modalities were addressed in time-space mapping studies. Results indicate that the majority of the studies supporting the asymmetry hypothesis applied visual tasks for both temporal and spatial representations. Studies supporting the symmetry hypothesis applied mainly auditory tasks for the temporal domain, but visual tasks for the spatial domain. We conclude that the use of different tasks addressing different modalities may be the primary reason for (a)symmetric effects of space on time, instead of a genuine (a)symmetric mapping.

Vision and Language Integration Meets Multimedia Fusion

Multimodal information fusion at both the signal and semantics level is a core part of most multimedia applications, including indexing, retrieval, and summarization. Prototype systems have implemented early or late fusion of modality-specific processing results through various methodologies including rule-based approaches, informationtheoretic models, and machine learning.1 Vision and language are two of the predominant modalities that are fused, with a long history of results in TRECVid, ImageClef, and other international challenges. During the last decade, vision– language semantic integration has attracted attention from traditionally non-interdisciplinary research communities such as computer vision and natural language processing. This is due to the fact that one modality can greatly assist in processing another by providing cues for disambiguation, complementary information, and noise/error filtering. Recent advances in deep learning have opened up new opportunities in joint modeling of visual and co-occurring verbal information in multimedia.

Modality-specific interference between action planning and perceptual processing

ABSTRACTThis paper reports a study which examined an interaction between action planning and processing of perceptual information in two different sensory modalities. In line with the idea that action planning consists in representing the action’s sensory outcomes, it was assumed that different types of actions should be coupled with different modalities. A visual and auditory oddball paradigm was combined with two types of actions: pointing and knocking (unrelated to the perceptual task). Results showed an interactive effect between the action type and the sensory modality of the oddballs, with impaired detection of auditory oddballs for knocking (congruent) action, as compared to a pointing (incongruent) action. These findings reveal that action planning can interact with modality-specific perceptual processing and that preparing an action presumably binds the respective perceptual features with an action plan, thereby making these features less available for other tasks.

Inside Speech: Multisensory and Modality-specific Processing of Tongue and Lip Speech Actions.

Action recognition has been found to rely not only on sensory brain areas but also partly on the observer's motor system. However, whether distinct auditory and visual experiences of an action modulate sensorimotor activity remains largely unknown. In the present sparse sampling fMRI study, we determined to which extent sensory and motor representations interact during the perception of tongue and lip speech actions. Tongue and lip speech actions were selected because tongue movements of our interlocutor are accessible via their impact on speech acoustics but not visible because of its position inside the vocal tract, whereas lip movements are both "audible" and visible. Participants were presented with auditory, visual, and audiovisual speech actions, with the visual inputs related to either a sagittal view of the tongue movements or a facial view of the lip movements of a speaker, previously recorded by an ultrasound imaging system and a video camera. Although the neural networks involved in visual visuolingual and visuofacial perception largely overlapped, stronger motor and somatosensory activations were observed during visuolingual perception. In contrast, stronger activity was found in auditory and visual cortices during visuofacial perception. Complementing these findings, activity in the left premotor cortex and in visual brain areas was found to correlate with visual recognition scores observed for visuolingual and visuofacial speech stimuli, respectively, whereas visual activity correlated with RTs for both stimuli. These results suggest that unimodal and multimodal processing of lip and tongue speech actions rely on common sensorimotor brain areas. They also suggest that visual processing of audible but not visible movements induces motor and visual mental simulation of the perceived actions to facilitate recognition and/or to learn the association between auditory and visual signals.

Visual Cortical Entrainment to Motion and Categorical Speech Features during Silent Lipreading.

Speech is a multisensory percept, comprising an auditory and visual component. While the content and processing pathways of audio speech have been well characterized, the visual component is less well understood. In this work, we expand current methodologies using system identification to introduce a framework that facilitates the study of visual speech in its natural, continuous form. Specifically, we use models based on the unheard acoustic envelope (E), the motion signal (M) and categorical visual speech features (V) to predict EEG activity during silent lipreading. Our results show that each of these models performs similarly at predicting EEG in visual regions and that respective combinations of the individual models (EV, MV, EM and EMV) provide an improved prediction of the neural activity over their constituent models. In comparing these different combinations, we find that the model incorporating all three types of features (EMV) outperforms the individual models, as well as both the EV and MV models, while it performs similarly to the EM model. Importantly, EM does not outperform EV and MV, which, considering the higher dimensionality of the V model, suggests that more data is needed to clarify this finding. Nevertheless, the performance of EMV, and comparisons of the subject performances for the three individual models, provides further evidence to suggest that visual regions are involved in both low-level processing of stimulus dynamics and categorical speech perception. This framework may prove useful for investigating modality-specific processing of visual speech under naturalistic conditions.

Multiple foci of spatial attention in multimodal working memory

The maintenance of sensory information in working memory (WM) is mediated by the attentional activation of stimulus representations that are stored in perceptual brain regions. Using event-related potentials (ERPs), we measured tactile and visual contralateral delay activity (tCDA/CDA components) in a bimodal WM task to concurrently track the attention-based maintenance of information stored in anatomically segregated (somatosensory and visual) brain areas. Participants received tactile and visual sample stimuli on both sides, and in different blocks, memorized these samples on the same side or on opposite sides. After a retention delay, memory was unpredictably tested for touch or vision. In the same side blocks, tCDA and CDA components simultaneously emerged over the same hemisphere, contralateral to the memorized tactile/visual sample set. In opposite side blocks, these two components emerged over different hemispheres, but had the same sizes and onset latencies as in the same side condition. Our results reveal distinct foci of tactile and visual spatial attention that were concurrently maintained on task-relevant stimulus representations in WM. The independence of spatially-specific biasing mechanisms for tactile and visual WM content suggests that multimodal information is stored in distributed perceptual brain areas that are activated through modality-specific processes that can operate simultaneously and largely independently of each other.

Effects of Line Separation and Exploration on the Visual and Haptic Detection of Symmetry and Repetition.

. Detection of regularities (e.g., symmetry, repetition) can be used to investigate object and shape perception. Symmetry and nearby lines may both signal that one object is present, so moving lines apart may disrupt symmetry detection, while repetition may signal that multiple objects are present. Participants discriminated symmetrical/irregular and repeated/irregular pairs of lines. For vision, as predicted, increased line separation disrupted symmetry detection more than repetition detection. For haptics, symmetry and repetition detection were similarly disrupted by increased line separation; also, symmetry was easier to detect than repetition for one-handed exploration and for body midline-aligned stimuli, whereas symmetry was harder to detect than repetition with two-handed exploration of stimuli oriented across the body. These effects of exploration and stimulus orientation show the influence of modality-specific processing rather than properties of the external world on regularity detection. These processes may, in turn, provide insights into the nature of objectness in vision and in touch.

Age-Related Declines in Early Sensory Memory: Identification of Rapid Auditory and Visual Stimulus Sequences.

Age-related temporal-processing declines of rapidly presented sequences may involve contributions of sensory memory. This study investigated recall for rapidly presented auditory (vowel) and visual (letter) sequences presented at six different stimulus onset asynchronies (SOA) that spanned threshold SOAs for sequence identification. Younger, middle-aged, and older adults participated in all tasks. Results were investigated at both equivalent performance levels (i.e., SOA threshold) and at identical physical stimulus values (i.e., SOAs). For four-item sequences, results demonstrated best performance for the first and last items in the auditory sequences, but only the first item for visual sequences. For two-item sequences, adults identified the second vowel or letter significantly better than the first. Overall, when temporal-order performance was equated for each individual by testing at SOA thresholds, recall accuracy for each position across the age groups was highly similar. These results suggest that modality-specific processing declines of older adults primarily determine temporal-order performance for rapid sequences. However, there is some evidence for a second amodal processing decline in older adults related to early sensory memory for final items in a sequence. This selective deficit was observed particularly for longer sequence lengths and was not accounted for by temporal masking.

Modalities of Thinking: State and Trait Effects on Cross-Frequency Functional Independent Brain Networks.

Functional states of the brain are constituted by the temporally attuned activity of spatially distributed neural networks. Such networks can be identified by independent component analysis (ICA) applied to frequency-dependent source-localized EEG data. This methodology allows the identification of networks at high temporal resolution in frequency bands of established location-specific physiological functions. EEG measurements are sensitive to neural activity changes in cortical areas of modality-specific processing. We tested effects of modality-specific processing on functional brain networks. Phasic modality-specific processing was induced via tasks (state effects) and tonic processing was assessed via modality-specific person parameters (trait effects). Modality-specific person parameters and 64-channel EEG were obtained from 70 male, right-handed students. Person parameters were obtained using cognitive style questionnaires, cognitive tests, and thinking modality self-reports. EEG was recorded during four conditions: spatial visualization, object visualization, verbalization, and resting. Twelve cross-frequency networks were extracted from source-localized EEG across six frequency bands using ICA. RMANOVAs, Pearson correlations, and path modelling examined effects of tasks and person parameters on networks. Results identified distinct state- and trait-dependent functional networks. State-dependent networks were characterized by decreased, trait-dependent networks by increased alpha activity in sub-regions of modality-specific pathways. Pathways of competing modalities showed opposing alpha changes. State- and trait-dependent alpha were associated with inhibitory and automated processing, respectively. Antagonistic alpha modulations in areas of competing modalities likely prevent intruding effects of modality-irrelevant processing. Considerable research suggested alpha modulations related to modality-specific states and traits. This study identified the distinct electrophysiological cortical frequency-dependent networks within which they operate.