Unimodal Task Research Articles

Multisensory subtypes of aphantasia: Mental imagery as supramodal perception in reverse

Cognitive neuroscience research on mental imagery has largely focused on the visual imagery modality in unimodal task contexts. Recent studies have uncovered striking individual differences in visual imagery capacity, with some individuals reporting a subjective absence of conscious visual imagery ability altogether (“aphantasia”). However, naturalistic mental imagery is often multi-sensory, and preliminary findings suggest that many individuals with aphantasia also report a subjective lack of mental imagery in other sensory domains (such as auditory or olfactory imagery). In this paper, we perform a series of cluster analyses on the multi-sensory imagery questionnaire scores of two large groups of aphantasic subjects, defining latent sub-groups in this sample population. We demonstrate that aphantasia is a heterogenous phenomenon characterised by dominant sub-groups of individuals with visual aphantasia (those who report selective visual imagery absence) and multi-sensory aphantasia (those who report an inability to generate conscious mental imagery in any sensory modality). We replicate our findings in a second large sample and show that more unique aphantasia sub-types also exist, such as individuals with selectively preserved mental imagery in only one sensory modality (e.g. intact auditory imagery). We outline the implications of our findings for network theories of mental imagery, discussing how unique aphantasia aetiologies with distinct self-report patterns might reveal alterations to various levels of the sensory processing hierarchy implicated in mental imagery.

Language and gesture neural correlates: A meta-analysis of functional magnetic resonance imaging studies.

Humans often use co-speech gestures to promote effective communication. Attention has been paid to the cortical areas engaged in the processing of co-speech gestures. To investigate the neural network underpinned in the processing of co-speech gestures and to observe whether there is a relationship between areas involved in language and gesture processing. We planned to include studies with neurotypical and/or stroke participants who underwent a bimodal task (i.e., processing of co-speech gestures with relative speech) and a unimodal task (i.e., speech or gesture alone) during a functional magnetic resonance imaging (fMRI) session. After a database search, abstract and full-text screening were conducted. Qualitative and quantitative data were extracted, and a meta-analysis was performed with the software GingerALE 3.0.2, performing contrast analyses of uni- and bimodal tasks. The database search produced 1024 records. After the screening process, 27 studies were included in the review. Data from 15 studies were quantitatively analysed through meta-analysis. Meta-analysis found three clusters with a significant activation of the left middle frontal gyrus and inferior frontal gyrus, and bilateral middle occipital gyrus and inferior temporal gyrus. There is a close link at the neural level for the semantic processing of auditory and visual information during communication. These findings encourage the integration of the use of co-speech gestures during aphasia treatment as a strategy to foster the possibility to communicate effectively for people with aphasia. What is already known on this subject Gestures are an integral part of human communication, and they may have a relationship at neural level with speech processing. What this paper adds to the existing knowledge During processing of bi- and unimodal communication, areas related to semantic processing and multimodal processing are activated, suggesting that there is a close link between co-speech gestures and spoken language at a neural level. What are the potential or actual clinical implications of this work? Knowledge of the functions related to gesture and speech processing neural networks will allow for the adoption of model-based neurorehabilitation programs to foster recovery from aphasia by strengthening the specific functions of these brain networks.

Correlation of motor-auditory cross-modal and auditory unimodal N1 and mismatch responses of schizophrenic patients and normal subjects: an MEG study.

It has been suggested that the positive symptoms of schizophrenic patients (hallucinations, delusions, and passivity experience) are caused by dysfunction of their internal and external sensory prediction errors. This is often discussed as related to dysfunction of the forward model that executes self-monitoring. Several reports have suggested that dysfunction of the forward model in schizophrenia causes misattributions of self-generated thoughts and actions to external sources. There is some evidence that the forward model can be measured using the electroencephalography (EEG) and magnetoencephalography (MEG) components such as N1 (m) and mismatch negativity (MMN) (m). The objective in this MEG study is to investigate differences in the N1m and MMNm-like activity generated in motor-auditory cross-modal tasks in normal control (NC) subjects and schizophrenic (SC) patients, and compared that activity with N1m and MMNm in the auditory unimodal task. The N1m and MMNm/MMNm-like activity were recorded in 15 SC patients and 12 matched NC subjects. The N1m-attenuation effects and peak amplitude of MMNm/MMNm-like activity of the NC and SC groups were compared. Additionally, correlations between MEG measures (N1m suppression rate, MMNm, and MMNm-like activity) and clinical variables (Positive and Negative Syndrome Scale (PANSS) scores and antipsychotic drug (APD) dosages) in SC patients were investigated. It was found that (i) there was no significant difference in N1m-attenuation for the NC and SC groups, and that (ii) MMNm in the unimodal task in the SC group was significantly smaller than that in the NC group. Further, the MMNm-like activity in the cross-modal task was smaller than that of the MMNm in the unimodal task in the NC group, but there was no significant difference in the SC group. The PANSS positive symptoms and general psychopathology score were moderately negatively correlated with the amplitudes of the MMNm-like activity, and the APD dosage was moderately negatively correlated with the N1m suppression rate. However, none of these correlations reached statistical significance. The findings suggest that schizophrenic patients perform altered predictive processes differently from healthy subjects in latencies reflecting MMNm, depending on whether they are under forward model generation or not. This may support the hypothesis that schizophrenic patients tend to misattribute their inner experience to external agents, thus leading to the characteristic schizophrenia symptoms.

MITER: Medical Image–TExt joint adaptive pretRaining with multi-level contrastive learning

Recently multimodal medical pretraining models play a significant role in automatic medical image and text analysis that has wide social and economical impact in healthcare. Despite being able to be quickly transferred to downstream tasks, the models are greatly limited due to the fact that these models can only be pretrained with professional medical image–text datasets, which usually contain a very small number of samples. In this work we propose MITER (Medical Image–Text Joint adaptive Pretraining), a joint adaptive pretraining framework via multi-level contrastive learning to overcome this limitation by pretraining image and text models for medical domain and utilizing existing models pretrained on generic data, which contain enormous number of samples. MITER features two types of objectives to solve the problem. The first type is uni-modal objectives that pretrain the models with medical images and text separately on uni-modal tasks. The other type is a cross-modal objective that pretrains jointly, allowing the models to influence each other on cross-modal tasks. We also introduce a strategy to dynamically select hard negative samples during the training process for better performance. Experimental results over four medical tasks, image-report retrieval, multi-label image classification, visual question answering, and report generation, show that our MITER framework solves the limitation problem by greatly outperforming existing benchmark models on all the tasks. The source code of our framework is available online.22https://github.com/ZhuYi98/MITER.

Multi-level correlation mining framework with self-supervised label generation for multimodal sentiment analysis

Fusion and co-learning are major challenges in multimodal sentiment analysis. Most existing methods either ignore the basic relationships among modalities or fail to maximize their potential correlations. They also do not leverage the knowledge from resource-rich modalities in the analysis of resource-poor modalities. To address these challenges, we propose a multimodal sentiment analysis method based on multilevel correlation mining and self-supervised multi-task learning. First, we propose a unimodal feature fusion- and linguistics-guided Transformer-based framework, multi-level correlation mining framework, to overcome the difficulty of multimodal information fusion. The module exploits the correlation information between modalities from low to high levels. Second, we divided the multimodal sentiment analysis task into one multimodal task and three unimodal tasks (linguistic, acoustic, and visual tasks), and designed a self-supervised label generation module (SLGM) to generate sentiment labels for unimodal tasks. SLGM-based multi-task learning overcomes the lack of unimodal labels in co-learning. Through extensive experiments on the CMU-MOSI and CMU-MOSEI datasets, we demonstrated the superiority of the proposed multi-level correlation mining framework to state-of-the-art methods.

Heterogeneous graph convolution based on In-domain Self-supervision for Multimodal Sentiment Analysis

The inability to fully exploit domain-specific knowledge and the lack of an effective integration method have been the difficulties and focus of multimodal sentiment analysis. In this paper, we propose heterogeneous graph convolution with in-domain self-supervised multi-task learning for multimodal sentiment analysis (HIS-MSA) to solve these problems. Firstly, HIS-MSA carries out the second pre-trained with different self-supervised training strategies to fully mine the unique knowledge of the in-domain corpus, and give BERT the awareness of professional field. Secondly, HIS-MSA uses heterogeneous graph, which is good at integrating heterogeneous knowledge, to fuse feature from multiple sources. Finally, a unimodal label generation module is used to jointly guide multimodal tasks and unimodal tasks to balance independent and complementary information between the modalities. We conducted experiments on the datasets MOSI and MOSEI, which have 2199 and 23454 video segments respectively. The results show an average improvement of approximately 1.5 points in all metrics compared to the current state-of-the-art model.

What Does a Language-And-Vision Transformer See: The Impact of Semantic Information on Visual Representations.

Neural networks have proven to be very successful in automatically capturing the composition of language and different structures across a range of multi-modal tasks. Thus, an important question to investigate is how neural networks learn and organise such structures. Numerous studies have examined the knowledge captured by language models (LSTMs, transformers) and vision architectures (CNNs, vision transformers) for respective uni-modal tasks. However, very few have explored what structures are acquired by multi-modal transformers where linguistic and visual features are combined. It is critical to understand the representations learned by each modality, their respective interplay, and the task’s effect on these representations in large-scale architectures. In this paper, we take a multi-modal transformer trained for image captioning and examine the structure of the self-attention patterns extracted from the visual stream. Our results indicate that the information about different relations between objects in the visual stream is hierarchical and varies from local to a global object-level understanding of the image. In particular, while visual representations in the first layers encode the knowledge of relations between semantically similar object detections, often constituting neighbouring objects, deeper layers expand their attention across more distant objects and learn global relations between them. We also show that globally attended objects in deeper layers can be linked with entities described in image descriptions, indicating a critical finding - the indirect effect of language on visual representations. In addition, we highlight how object-based input representations affect the structure of learned visual knowledge and guide the model towards more accurate image descriptions. A parallel question that we investigate is whether the insights from cognitive science echo the structure of representations that the current neural architecture learns. The proposed analysis of the inner workings of multi-modal transformers can be used to better understand and improve on such problems as pre-training of large-scale multi-modal architectures, multi-modal information fusion and probing of attention weights. In general, we contribute to the explainable multi-modal natural language processing and currently shallow understanding of how the input representations and the structure of the multi-modal transformer affect visual representations.

Rethinking attentional reset: Task sets determine the boundaries of adaptive control.

Adaptive control processes that minimise distraction often operate in a context-specific manner. For example, they may minimise distraction from irrelevant conversations during a lecture but not in the hallway afterwards. It remains unclear, however, whether (a) salient perceptual features or (b) task sets based on such features serve as contextual boundaries for adaptive control in standard distractor-interference tasks. To distinguish between these possibilities, we manipulated whether the structure of a standard, visual distractor-interference task allowed (Experiment 1) or did not allow (Experiment 2) participants to associate salient visual features (i.e., colour patches and colour words) with different task sets. We found that changing salient visual features across consecutive trials reduced a popular measure of adaptive control in distractor-interference tasks-the congruency sequence effect (CSE)-only when the task structure allowed participants to associate these visual features with different task sets. These findings extend prior support for the task set hypothesis from somewhat atypical cross-modal tasks to a standard unimodal task. In contrast, they pose a challenge to an alternative "attentional reset" hypothesis, and related views, wherein changing salient perceptual features always results in a contextual boundary for the CSE.

Promoting neuroplasticity and neuropsychological functioning in frailty through an app-based sensorimotor training: study protocol for a randomized trial

BackgroundFrailty is characterized by an age-related decline in multiple physiological systems, leading to a high vulnerability to stressors, adverse health outcomes, and low quality of life. Neuroscientific models of pathological aging emphasize the loss of sensorimotor stimulation and reduced neuromodulatory capacities as core processes in age-related cognitive and bodily decline, which may be associated with maladaptive plastic changes in the brain. We plan to increase sensorimotor stimulation in frail persons through a newly developed app-based training program and link the training trials to biological and psychological correlates of age-associated vulnerability and health indices.MethodsWe will conduct a randomized trial, applying an app-based sensorimotor home training (N = 30) in people suffering from frailty. An app-based relaxation training will serve as an active control condition (N = 30). Both interventions will last for 90 days each. The sensorimotor training includes unimodal and multimodal sensory discrimination tasks in the visual, auditory, and tactile domain, as well as sensorimotor precision tasks. The tasks will be implemented using an adaptive training algorithm and enriched with motivational components embedded in a virtual training environment. We expect a pre-post reduction of frailty status and associated functional decline related to refinement of representational maps within the sensorimotor system and improved sensorimotor function such as extremity function. Secondary analyses will study the influence of BDNF genotype as moderating variable. Additional outcomes will include measures of perceptual and cognitive functioning, quality of life as well as BDNF serum levels. Measurements will take place before training (baseline), after 60 days (assessment 1), and at the end of the training after 90 days (assessment 2).DiscussionIn our randomized trial, we aim to characterize a multidimensional concept of frailty and to target maladaptive behaviors and neuroplasticity using an app-based sensorimotor training. This type of intervention might provide further knowledge and new possibilities for preventing decline and preserving function in older adults.Trial registrationClinicalTrials.gov NCT03666039. Registered 11 September 2018 – Retrospectively registered.Protocol version: Version 4 revised (issue date: 19 May 2021).

Regularizing Attention Networks for Anomaly Detection in Visual Question Answering

For stability and reliability of real-world applications, the robustness of DNNs in unimodal tasks has been evaluated. However, few studies consider abnormal situations that a visual question answering (VQA) model might encounter at test time after deployment in the real-world. In this study, we evaluate the robustness of state-of-the-art VQA models to five different anomalies, including worst-case scenarios, the most frequent scenarios, and the current limitation of VQA models. Different from the results in unimodal tasks, the maximum confidence of answers in VQA models cannot detect anomalous inputs, and post-training of the outputs, such as outlier exposure, is ineffective for VQA models. Thus, we propose an attention-based method, which uses confidence of reasoning between input images and questions and shows much more promising results than the previous methods in unimodal tasks. In addition, we show that a maximum entropy regularization of attention networks can significantly improve the attention-based anomaly detection of the VQA models. Thanks to the simplicity, attention-based anomaly detection and the regularization are model-agnostic methods, which can be used for various cross-modal attentions in the state-of-the-art VQA models. The results imply that cross-modal attention in VQA is important to improve not only VQA accuracy, but also the robustness to various anomalies.

Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis

Representation Learning is a significant and challenging task in multimodal learning. Effective modality representations should contain two parts of characteristics: the consistency and the difference. Due to the unified multimodal annota- tion, existing methods are restricted in capturing differenti- ated information. However, additional unimodal annotations are high time- and labor-cost. In this paper, we design a la- bel generation module based on the self-supervised learning strategy to acquire independent unimodal supervisions. Then, joint training the multimodal and uni-modal tasks to learn the consistency and difference, respectively. Moreover, dur- ing the training stage, we design a weight-adjustment strat- egy to balance the learning progress among different sub- tasks. That is to guide the subtasks to focus on samples with the larger difference between modality supervisions. Last, we conduct extensive experiments on three public multimodal baseline datasets. The experimental results validate the re- liability and stability of auto-generated unimodal supervi- sions. On MOSI and MOSEI datasets, our method surpasses the current state-of-the-art methods. On the SIMS dataset, our method achieves comparable performance than human- annotated unimodal labels. The full codes are available at https://github.com/thuiar/Self-MM.

A bimodal extension of the Eriksen flanker task

The Eriksen flanker task is a traditional conflict paradigm for studying the influence of task-irrelevant information on the processing of task-relevant information. In this task, participants are asked to respond to a visual target item (e.g., a letter) that is flanked by task-irrelevant items (e.g., also letters). Responses are typically faster and more accurate when the task-irrelevant information is response-congruent with the visual target than when it is incongruent. Several researchers have attributed the starting point of this flanker effect to poor selective filtering at a perceptual level (e.g., spotlight models), which subsequently produces response competition at post-perceptual stages. The present study examined whether a flanker-like effect could also be established within a bimodal analog of the flanker task with auditory irrelevant letters and visual target letters, which must be processed along different processing routes. The results of two experiments revealed that a flanker-like effect is also present with bimodal stimuli. In contrast to the unimodal flanker task, however, the effect only emerged when flankers and targets shared the same letter name, but not when they were different letters mapped onto the same response. We conclude that the auditory flankers can influence the time needed to recognize visual targets but do not directly activate their associated responses.

Monitoring the Clinical Evolution of A Psychotic Patient Presenting A First-Schizophrenic Episode Thanks to Bimodal Oddball-P300 Event-Related Potentials: First Evidence from A Single-Case Study

Objective: Previous studies showed that a bimodal oddball design allows to increase the sensitivity of the P300 in subjects with anxious-depressive tendencies and patients presenting an adjustment disorder. In this case report we illustrate the increased sensitivity of a bimodal oddball task in the clinical follow-up of a patient presenting a psychosis in the context of a first onset schizophrenia. Method: The patient was instructed to detect deviant stimuli in three separate oddball tasks by hitting a button. Tasks comprised an auditory, a visual and a synchronized and congruent audio-visual block of stimuli. The peak-latency and the amplitude of the P300 evoked response were compared and confronted to scores of the clinical assessment. Results: Only the bimodal task mirrored the clinical evolution of the patient, as the reduction of psychotic symptoms was accompanied by an increase of the bimodal P300 amplitude. However, this was not true for both unimodal tasks (auditory and visual). Conclusions: A bimodal oddball design combined with unimodal ones could be necessary to monitor the joined clinical/neurophysiological evolution of a patient presenting a first onset psychosis. Significance: Further longitudinal studies applying the bimodal P300 paradigm should be designed in order to verify whether the oddball-bimodal P300 component could be used as “state” and/ or “trait” biological marker of the disease.

Altered Auditory Self-recognition in People with Schizophrenia.

Self-recognition is of great significance to our sense of self. To date, disturbances in the processing of visual self-recognition are well studied in people with schizophrenia, whereas relatively few studies have focused on the processing of self in other domains, such as auditory. An investigation of auditory self-recognition contributes to delineate changes related to self and the potential roots of the described psychopathological aspects connoting schizophrenia. By applying unimodal task and multisensory test, this study investigated auditory self-recognition in people with schizophrenia under unimodal and bimodal circumstances. Forty-six adults diagnosed with schizophrenia and thirty-two healthy controls were involved in this study. Results suggested that people with schizophrenia seemed to have significantly lower perceptual sensitivity in detecting self-voice, and also showed stricter judgment criteria in self-voice decision. Furthermore, in the presentation of stimuli that combined the stimulation of others' faces with one's own voice, people with schizophrenia mistakenly attributed the voices of others as their own. In conclusion, altered auditory self-recognition in people with schizophrenia was found.

A multi-strategy enhanced sine cosine algorithm for global optimization and constrained practical engineering problems

The Sine Cosine Algorithm (SCA) has received much attention from engineering and scientific fields since it was proposed. Nevertheless, when solving multimodal or complex high dimensional optimization tasks, the conventional SCA still has a high probability of falling into the local optimal stagnation or failing to obtain the global optimum solution. Additionally, it performspoorly in convergence. Therefore, in this study, a multi-strategy enhanced SCA, a memetic algorithm termed MSCA, is proposed, which combines multiple control mechanisms including Cauchy mutation operator, chaotic local search mechanism, opposition-based learning strategy and two operators based on differential evolution to achieve a better balance between exploration and exploitation. To verify its performance, MSCA was compared with 11 state-of-the-art original optimizers and variant algorithms on 23 continuous benchmark tasks including 7 unimodal tasks, 6 multimodal tasks, 10 various fixed-dimension multimodal functions, and several typical CEC2014 benchmark problems. Furthermore, MSCA was utilized to solve three constrained practical engineering problems including tension/compression spring design, welded beam design, and pressure vessel design. The experimental results demonstrate that the proposed algorithm MSCA is superior to other competitors in terms of quality of solutions and convergence speed and can serve as an effective andefficient computer-aided tool for practical tasks with complex search space.

Multimodal Integration and Vividness in the Angular Gyrus During Episodic Encoding and Retrieval.

Much evidence suggests that the angular gyrus (AnG) is involved in episodic memory, but its precise role has yet to be determined. We examined two possible accounts within the same experimental paradigm: the “cortical binding of relational activity” (CoBRA) account (Shimamura, 2011), which suggests that the AnG acts as a convergence zone that binds multimodal episodic features, and the subjectivity account (Yazar et al., 2012), which implicates AnG involvement in subjective mnemonic experience (such as vividness or confidence). fMRI was used during both encoding and retrieval of paired associates. During study, female and male human participants memorized picture-pairs of common objects (in the unimodal task) or of an object-picture and an environmental sound (in the crossmodal task). At test, they performed a cued-recall task and further indicated the vividness of their memory. During retrieval, BOLD activation in the AnG was greatest for vividly remembered associates, consistent with the subjectivity account. During encoding, the same effect of vividness was found, but this was further modulated by task: greater activations were associated with subsequent recall in the crossmodal than the unimodal task. Therefore, encoding data suggest an additional role to the AnG in crossmodal integration, consistent with its role at retrieval proposed by CoBRA. These results resolve some of the puzzles in the literature and indicate that the AnG can play different roles during encoding and retrieval as determined by the cognitive demands posed by different mnemonic tasks.SIGNIFICANCE STATEMENT We offer new insights into the multiplicity of processes that are associated with angular gyrus (AnG) activation during encoding and retrieval of newly formed memories. We used fMRI while human participants learned and subsequently recalled pairs of objects presented to the same sensory modality or to different modalities. We were able to show that the AnG is involved when vivid memories are created and retrieved, as well as when encoded information is integrated across different sensory modalities. These findings provide novel evidence for the contribution of the AnG to our subjective experience of remembering alongside its role in integrative processes that promote subsequent memory.

Robust Temporal Averaging of Time Intervals Between Action and Sensation.

Perception of the time interval between one’s own action (a finger tapping) and the associated sensory feedback (a visual flash or an auditory beep) is critical for precise and flexible control of action and behavioral decision. Previous studies have examined temporal averaging for multiple time intervals and its role for perceptual organization and crossmodal integration. In the present study, we extended the temporal averaging from sensory stimuli to the coupling of action and its sensory feedback. We investigated whether and how temporal averaging could be achieved with respect to the multiple intervals in a sequence of action-sensory feedback events, and hence affect the subsequent timing behavior. In unimodal task, participants voluntarily tapped their index finger at a constant pace while receiving auditory feedback (beeps) with varied intervals as well as variances throughout the sequence. In crossmodal task, for a given sequence, each tap was accompanied randomly with either visual flash or auditory beep as sensory feedback. When the sequence was over, observers produced a subsequent tap with either auditory or visual stimulus, which enclose a probe interval. In both tasks, participants were required to make a two alternative forced choice (2AFC), to indicate whether the target interval is shorter or longer than the mean interval between taps and their associated sensory events in the preceding sequence. In both scenarios, participants’ judgments of the probe interval suggested that they had internalized the mean interval associated with specific bindings of action and sensation, showing a robust temporal averaging process for the interval between action and sensation.

The P300 component decreases in a bimodal oddball task in individuals with depression: An event-related potentials study

ObjectiveIn this study, we investigated auditory-visual stimulation-induced P300 and examined whether P300 was differentially modulated between individuals with clinical depression and healthy controls. We hypothesized that the P300 component would significantly differ between individuals with depression and healthy individuals Specifically, we predicted that the P300 component induced by the bimodal oddball task would be significantly different from that induced by the unimodal task. MethodsForty-five individuals with depression and forty-five healthy controls participated in this study. All participants were instructed to complete three oddball tasks—auditory (A), visual (V), and bimodal (AV)—while their electroencephalographic signals were recorded. ResultsIndividuals with depression had a lower P300 amplitude and a longer latency than controls in the bimodal task. P300 amplitudes in the bimodal task were significantly higher than in the auditory or visual tasks in both groups. In the depression group, the P300 amplitude was negatively correlated with Hamilton Depression Rating Scale (HAM-D) scores in the bimodal task. ConclusionsOur results, which agree with those reported previously, suggest that there is a heightened P300 amplitude sensitivity in the bimodal task in individuals with depression. Our data also suggest that P300 amplitudes in the bimodal task may reflect the severity of depression. SignificanceThe reduced task-related ERP response in individuals with depression suggests significant impairments in these individuals in stimulus integration and response functions.

Neural Correlates of Feedback Processing in Visuo-Tactile Crossmodal Paired-Associate Learning.

Previous studies have examined the neural correlates for crossmodal paired-associate (PA) memory and the temporal dynamics of its formation. However, the neural dynamics for feedback processing of crossmodal PA learning remain unclear. To examine this process, we recorded event-related scalp electrical potentials for PA learning of unimodal visual-visual pairs and crossmodal visual-tactile pairs when participants performed unimodal and crossmodal tasks. We examined event-related potentials (ERPs) after the onset of feedback in the tasks for three effects: feedback type (positive feedback vs. negative feedback), learning (as the learning progressed) and the task modality (crossmodal vs. unimodal). The results were as follows: (1) feedback type: the amplitude of P300 decreased with incorrect trials and the P400/N400 complex was only present in incorrect trials; (2) learning: progressive positive voltage shifts in frontal recording sites and negative voltage shifts in central and posterior recording sites were identified as learning proceeded; and (3) task modality: compared with the unimodal PA learning task, positive voltage shifts in frontal sites and negative voltage shifts in posterior sites were found in the crossmodal PA learning task. To sum up, these results shed light on cortical excitability related to feedback processing of crossmodal PA learning.

Information processes of task-switching and modality-shifting across development.

Developmental research on flexible attentional control in young children has often focused on the role of attention in task-switching in a unimodal context. In real life, children must master the art of switching attention not only between task demands, but also between sensory modalities. Previous study has shown that young children can be efficient at switching between unimodal tasks when the situation allows, incurring no greater task-switching costs than adults. However, young children may still experience a greater demand to shift attention between modalities than older participants. To address this, we tested 4-year-olds, 6-year-olds and adults on a novel cross-modal task-switching paradigm involving multisensory detection tasks. While we found age differences in absolute reaction time and accuracy, young children and adults both exhibited strikingly similar effects in task-switching, modality-shifting, and the interaction between them. Young children did not exhibit a greater attentional bottleneck on either the task level, or on the modality level; thus, the evidence suggests that young children engaged in similar cognitive operations in the current cross-modal tasks to adult participants. It appears that cognitive operations in multisensory task configuration are relatively mature between 4 and 6 years old.