Cross-modal Connections Research Articles

Concept-Guided Prompt Learning for Generalization in Vision-Language Models

Contrastive Language-Image Pretraining (CLIP) model has exhibited remarkable efficacy in establishing cross-modal connections between texts and images, yielding impressive performance across a broad spectrum of downstream applications through fine-tuning. However, for generalization tasks, the current fine-tuning methods for CLIP, such as CoOp and CoCoOp, demonstrate relatively low performance on some fine-grained datasets. We recognize the underlying reason is that these previous methods only projected global features into the prompt, neglecting the various visual concepts, such as colors, shapes, and sizes, which are naturally transferable across domains and play a crucial role in generalization tasks. To address this issue, in this work, we propose Concept-Guided Prompt Learning (CPL) for vision-language models. Specifically, we leverage the well-learned knowledge of CLIP to create a visual concept cache to enable conceptguided prompting. In order to refine the text features, we further develop a projector that transforms multi-level visual features into text features. We observe that this concept-guided prompt learning approach is able to achieve enhanced consistency between visual and linguistic modalities. Extensive experimental results demonstrate that our CPL method significantly improves generalization capabilities compared to the current state-of-the-art methods.

Self-Supervised Video Representation and Temporally Adaptive Attention for Audio-Visual Event Localization

Localizing the audio-visual events in video requires a combined judgment of visual and audio components. To integrate multimodal information, existing methods modeled the cross-modal relationships by feeding unimodal features into attention modules. However, these unimodal features are encoded in separate spaces, resulting in a large heterogeneity gap between modalities. Existing attention modules, on the other hand, ignore the temporal asynchrony between vision and hearing when constructing cross-modal connections, which may lead to the misinterpretation of one modality by another. Therefore, this paper aims to improve event localization performance by addressing these two problems and proposes a framework that feeds audio and visual features encoded in the same semantic space into a temporally adaptive attention module. Specifically, we develop a self-supervised representation method to encode features with a smaller heterogeneity gap by matching corresponding semantic cues between synchronized audio and visual signals. Furthermore, we develop a temporally adaptive cross-modal attention based on a weighting method that dynamically channels attention according to the time differences between event-related features. The proposed framework achieves state-of-the-art performance on the public audio-visual event dataset and the experimental results not only show that our self-supervised method can learn more discriminative features but also verify the effectiveness of our strategy for assigning attention.

Case Report: Neural correlates of orthographic congruency effects in auditory rhyme judgment in two stroke survivors

ABSTRACT Background Auditory rhyme judgment has been widely used as an index of phonological processing in people with aphasia (PWA). However, healthy, literate adults experience orthographic interference during phonological tasks such as auditory rhyme judgment, a byproduct of skilled reading and crossmodal connections forged during reading acquisition. This orthographic interference is an indicator that orthographic representations are active during phonological processing. Therefore, the potential for orthographic interference in auditory rhyme judgments in PWA may indicate that performance on such tasks is supported by stored graphophonemic representations, rather than specifically indexing phonological processing. Method Using event related potentials (ERPs), we examined orthographic congruency effects in auditory rhyme judgment in two stroke survivors, one with moderate nonfluent aphasia and apraxia of speech, and one with mild nonfluent aphasia and apraxia of speech. The two stroke survivors and nine healthy comparison participants completed an auditory rhyme judgment task in four conditions (rhyming and non-rhyming, with congruent and incongruent orthography). Results ERP results showed the N400 rhyming and orthographic congruency effects for both stroke survivors and the comparison group even though words were not presented visually. Conclusion The results of this case report indicate that orthographic representations are active during auditory rhyme judgment in the two stroke survivors, similar to comparison participants. Thus, auditory rhyme judgment tasks may be supported by orthographic representations and therefore may not index solely phonological processing, but rather crossmodal graphophonemic representations.

Induction Mechanism of Auditory-Assisted Vision for Target Search Localization in Mixed Reality (MR) Environments

In MR (mixed reality) environments, visual searches are often used for search and localization missions. There are some problems with search and localization technologies, such as a limited field of view and information overload. They are unable to satisfy the need for the rapid and precise location of specific flying objects in a group of air and space targets under modern air and space situational requirements. They lead to inefficient interactions throughout the mission process. A human being’s decision and judgment will be affected by inefficient interactions. Based on this problem, we carried out a multimodal optimization study on the use of an auditory-assisted visual search for localization in an MR environment. In the spatial–spherical coordinate system, the target flight object position is uniquely determined by the height h, distance r, and azimuth θ. Therefore, there is an urgent need to study the cross-modal connections between the auditory elements and these three coordinates based on a visual search. In this paper, an experiment was designed to study the correlation between auditory intuitive perception and vision and the cognitive induction mechanism. The experiment included the three cross-modal mappings of pitch–height, volume–distance, and vocal tract alternation–spatial direction. The research conclusions are as follows: (1) Visual cognition is induced by high, medium, and low pitches to be biased towards the high, medium, and low spatial regions of the visual space. (2) Visual cognition is induced by loud, medium, and low volumes to be biased towards the near, middle, and far spatial regions of the visual space. (3) Based on the HRTF application, the vocal track alternation scheme is expected to significantly improve the efficiency of visual interactions. Visual cognition is induced by left short sounds, right short sounds, left short and long sounds, and right short and long sounds to be biased towards the left, right, left-rear, and right-rear directions of visual space. (4) The cognitive load of search and localization technologies is significantly reduced by incorporating auditory factors. In addition, the efficiency and effect of the accurate search and positioning of space-flying objects have been greatly improved. The above findings can be applied to the research on various types of target search and localization technologies in an MR environment and can provide a theoretical basis for the subsequent study of spatial information perception and cognitive induction mechanisms in an MR environment with visual–auditory coupling.

Excitatory Crossmodal Input to a Widespread Population of Primary Sensory Cortical Neurons.

Crossmodal information processing in sensory cortices has been reported in sparsely distributed neurons under normal conditions and can undergo experience- or activity-induced plasticity. Given the potential role in brain function as indicated by previous reports, crossmodal connectivity in the sensory cortex needs to be further explored. Using perforated whole-cell recording in anesthetized adult rats, we found that almost all neurons recorded in the primary somatosensory, auditory, and visual cortices exhibited significant membrane-potential responses to crossmodal stimulation, as recorded when brain activity states were pharmacologically down-regulated in light anesthesia. These crossmodal cortical responses were excitatory and subthreshold, and further seemed to be relayed primarily by the sensory thalamus, but not the sensory cortex, of the stimulated modality. Our experiments indicate a sensory cortical presence of widespread excitatory crossmodal inputs, which might play roles in brain functions involving crossmodal information processing or plasticity.

A cortical circuit for audio-visual predictions

Learned associations between stimuli in different sensory modalities can shape the way we perceive these stimuli. However, it is not well understood how these interactions are mediated or at what level of the processing hierarchy they occur. Here we describe a neural mechanism by which an auditory input can shape visual representations of behaviorally relevant stimuli through direct interactions between auditory and visual cortices in mice. We show that the association of an auditory stimulus with a visual stimulus in a behaviorally relevant context leads to experience-dependent suppression of visual responses in primary visual cortex (V1). Auditory cortex axons carry a mixture of auditory and retinotopically matched visual input to V1, and optogenetic stimulation of these axons selectively suppresses V1 neurons that are responsive to the associated visual stimulus after, but not before, learning. Our results suggest that cross-modal associations can be communicated by long-range cortical connections and that, with learning, these cross-modal connections function to suppress responses to predictable input.

Cross-modal connectivity effects in age-related hearing loss

Age-related sensorineural hearing loss (HL) leads to localized brain changes in the primary auditory cortex, long-range functional alterations, and is considered a risk factor for dementia. Nonhuman studies have repeatedly highlighted cross-modal brain plasticity in sensorial brain networks other than those primarily involved in the peripheral damage, thus in this study, the possible cortical alterations associated with HL have been analyzed using a whole-brain multimodal connectomic approach.Fifty-two HL and 30 normal hearing participants were examined in a 3T MRI study along with audiological and neurological assessments. Between-regions functional connectivity and whole-brain probabilistic tractography were calculated in a connectome-based manner and graph theory was used to obtain low-dimensional features for the analysis of brain connectivity at global and local levels.The HL condition was associated with a different functional organization of the visual subnetwork as revealed by a significant increase in global efficiency, density, and clustering coefficient. These functional effects were mirrored by similar (but more subtle) structural effects suggesting that a functional repurposing of visual cortical centers occurs to compensate for age-related loss of hearing abilities.

Odors in Cognitive Research: a commentary on 'Scented Colours' and an evaluation study on odor quality, with the example of human wayfinding

In his target article on “Scented Colours”, Charles Spence highlights the importance of crossmodal connections by focusing on the interaction between odors and colors. In this commentary and our presentation of own empirical work in this research context, we want to reach out further by emphasizing this importance not only on a perceptual and representational level, but also highlight it as an example for spatial cognition research. We provide an evaluation study on emotional effects of odors that could be used in future interdisciplinary research. While the meaning of odors in spatial wayfinding is, thus far, not well investigated, we want to briefly introduce the current research state, focus on new and valuable research strains in this area as an example for the importance of odors in cognitive research.

Neuroplasticity and crossmodal connectivity in the normal, healthy brain.

Neuroplasticity enables the brain to establish new crossmodal connections or reorganize old connections which are essential to perceiving a multisensorial world. The intent of this review is to identify and summarize the current developments in neuroplasticity and crossmodal connectivity, and deepen understanding of how crossmodal connectivity develops in the normal, healthy brain, highlighting novel perspectives about the principles that guide this connectivity. To the above end, a narrative review is carried out. The data documented in prior relevant studies in neuroscience, psychology and other related fields available in a wide range of prominent electronic databases are critically assessed, synthesized, interpreted with qualitative rather than quantitative elements, and linked together to form new propositions and hypotheses about neuroplasticity and crossmodal connectivity. Three major themes are identified. First, it appears that neuroplasticity operates by following eight fundamental principles and crossmodal integration operates by following three principles. Second, two different forms of crossmodal connectivity, namely direct crossmodal connectivity and indirect crossmodal connectivity, are suggested to operate in both unisensory and multisensory perception. Third, three principles possibly guide the development of crossmodal connectivity into adulthood. These are labeled as the principle of innate crossmodality, the principle of evolution-driven 'neuromodular' reorganization and the principle of multimodal experience. These principles are combined to develop a three-factor interaction model of crossmodal connectivity. The hypothesized principles and the proposed model together advance understanding of neuroplasticity, the nature of crossmodal connectivity, and how such connectivity develops in the normal, healthy brain.

The Neurological Basis of Developmental Dyslexia and Related Disorders: A Reappraisal of the Temporal Hypothesis, Twenty Years on.

In a now-classic article published a couple of decades ago (Brain, 2000; 123: 2373–2399), I proposed an “extended temporal processing deficit hypothesis of dyslexia”, suggesting that a deficit in temporal processing could explain not only language-related peculiarities usually noticed in dyslexic children, but also a wider range of symptoms related to impaired processing of time in general. In the present review paper, I will revisit this “historical” hypothesis both in the light of a new clinical perspective, including the central yet poorly explained notion of comorbidity, and also taking a new look at the most recent experimental work, mainly focusing on brain imaging data. First, consistent with daily clinical practice, I propose to distinguish three groups of children who fail to learn to read, of fairly equal occurrence, who share the same initial presentation (difficulty in mastering the rules of grapheme–phoneme correspondence) but with differing associated signs and/or comorbid conditions (language disorders in the first group, attentional deficits in the second one, and motor coordination problems in the last one), thus suggesting, at least in part, potentially different triggering mechanisms. It is then suggested, in the light of brain imaging information available to date, that the three main clinical presentations/associations of cognitive impairments that compromise reading skills acquisition correspond to three distinct patterns of miswiring or “disconnectivity” in specific brain networks which have in common their involvement in the process of learning and their heavy reliance on temporal features of information processing. With reference to the classic temporal processing deficit of dyslexia and to recent evidence of an inability of the dyslexic brain to achieve adequate coupling of oscillatory brain activity to the temporal features of external events, a general model is proposed according to which a common mechanism of temporal uncoupling between various disconnected—and/or mis-wired—processors may account for distinct forms of specific learning disorders, with reading impairment being a more or less constant feature. Finally, the potential therapeutic implications of such a view are considered, with special emphasis on methods seeking to enhance cross-modal connectivity between separate brain systems, including those using rhythmic and musical training in dyslexic patients.

TIME: Text and Image Mutual-Translation Adversarial Networks

Focusing on text-to-image (T2I) generation, we propose Text and Image Mutual-Translation Adversarial Networks (TIME), a lightweight but effective model that jointly learns a T2I generator G and an image captioning discriminator D under the Generative Adversarial Network framework. While previous methods tackle the T2I problem as a uni-directional task and use pre-trained language models to enforce the image--text consistency, TIME requires neither extra modules nor pre-training. We show that the performance of G can be boosted substantially by training it jointly with D as a language model. Specifically, we adopt Transformers to model the cross-modal connections between the image features and word embeddings, and design an annealing conditional hinge loss that dynamically balances the adversarial learning. In our experiments, TIME achieves state-of-the-art (SOTA) performance on the CUB dataset (Inception Score of 4.91 and Fréchet Inception Distance of 14.3 on CUB), and shows promising performance on MS-COCO dataset on image captioning and downstream vision-language tasks.

Predictive Coding Over the Lifespan: Increased Reliance on Perceptual Priors in Older Adults-A Magnetoencephalography and Dynamic Causal Modeling Study.

Aging is accompanied by unisensory decline. To compensate for this, two complementary strategies are potentially relied upon increasingly: first, older adults integrate more information from different sensory organs. Second, according to the predictive coding (PC) model, we form “templates” (internal models or “priors”) of the environment through our experiences. It is through increased life experience that older adults may rely more on these templates compared to younger adults. Multisensory integration and predictive coding would be effective strategies for the perception of near-threshold stimuli, which may however come at the cost of integrating irrelevant information. Both strategies can be studied in multisensory illusions because these require the integration of different sensory information, as well as an internal model of the world that can take precedence over sensory input. Here, we elicited a classic multisensory illusion, the sound-induced flash illusion, in younger (mean: 27 years, N = 25) and older (mean: 67 years, N = 28) adult participants while recording the magnetoencephalogram. Older adults perceived more illusions than younger adults. Older adults had increased pre-stimulus beta-band activity compared to younger adults as predicted by microcircuit theories of predictive coding, which suggest priors and predictions are linked to beta-band activity. Transfer entropy analysis and dynamic causal modeling of pre-stimulus magnetoencephalography data revealed a stronger illusion-related modulation of cross-modal connectivity from auditory to visual cortices in older compared to younger adults. We interpret this as the neural correlate of increased reliance on a cross-modal predictive template in older adults leading to the illusory percept.

STDP Based Unsupervised Multimodal Learning With Cross-Modal Processing in Spiking Neural Networks

Spiking neural networks perform reasonably well in recognition applications for single modality (e.g., images, audio, or text). In this paper, we propose a multimodal spiking neural network that combines two modalities (image and audio). The two unimodal ensembles are connected with cross-modal connections and the entire network is trained with unsupervised learning. The network receives inputs in both modalities for the same class and predicts the class label. The excitatory connections in the unimodal ensemble and the cross-modal connections are trained with power-law weight-dependent spike timing dependent plasticity learning rule. The cross-modal connections capture the correlation between neurons of different modalities. The multimodal network learns features of both modalities and improves the classification accuracy compared to unimodal topology, even when one of the modality is distorted by noise. The cross-modal connections suppress the effect of noise on classification accuracy. The well-learned cross-modal connections invoke additional spiking activity in neurons of the correct label. The cross-modal connections are only excitatory and do not inhibit the normal activity of the unimodal ensembles. We evaluated our multimodal network on images from MNIST dataset and utterances of digits from TI46 speech corpus. The multimodal network achieved a classification accuracy of 98% on the combined MNIST and TI46 dataset.

Unsupervised Machine Learning Based Scalable Fusion for Active Perception

The fusion of sensor data is an essential requirement for active perception in autonomous robots in order to accurately perceive an environment, make appropriate adjustments that improve the understanding of the environment and generate actionable insights. Despite a large body of literature in this domain, only a handful of research focuses on unsupervised machine learning, which is increasingly important in unlabeled, unknown environments. This paper proposes a scalable self-organizing neural architecture for environmental perception based on multimodal fusion using unsupervised machine learning. Inspired by the biological counterpart, the neural architecture consists of topographic maps across each modality and incorporates a novel scalable self-organization process that handles high volume, high-velocity sensor data. The co-occurrence relationships across modalities are captured in cross-modal connections and the reverse Hebbian projection is used for multimodal representation. This proposed unsupervised machine learning-based scalable fusion technique was implemented on Apache Hadoop and Apache Spark. It was validated using an extensive multimodal human activity sensor dataset to demonstrate efficient representation and multimodal fusion for active perception.

Für eine Onto-Ästhetik des Autismus

The present paper analyses synesthesia in autism (ASD) and neurotypical experience. The way synesthesia has been interpreted within the history of phenomenology and psychopathology should prepare a philosophical access to autism and its subjective condition. The paper draws on the basic assumption that synesthesia reveals the presence of sensible networks and crossmodal connections beneath the framework of objective reality. Synesthesia, it is argued, confronts us with a specific dimension of the world beneath the essential structures of a material apriori, as it is elaborated within husserlian phenomenology. Experience in autism has close connections with such an alternative ontological setting. Asa matter of fact, autistic self-advocates often describe moments of a deep immersion into sensible experience, in which different sensorial events come to ‘resonate’ with one another. Resonance is thus interpreted as a fundamental ontological connector within an experiential framework that is not subjected to abstract concepts and causal thinking

Crossmodal Connections of Primary Sensory Cortices Largely Vanish During Normal Aging.

During aging, human response times (RTs) to unisensory and crossmodal stimuli decrease. However, the elderly benefit more from crossmodal stimulus representations than younger people. The underlying short-latency multisensory integration process is mediated by direct crossmodal connections at the level of primary sensory cortices. We investigate the age-related changes of these connections using a rodent model (Mongolian gerbil), retrograde tracer injections into the primary auditory (A1), somatosensory (S1), and visual cortex (V1), and immunohistochemistry for markers of apoptosis (Caspase-3), axonal plasticity (Growth associated protein 43, GAP 43), and a calcium-binding protein (Parvalbumin, PV). In adult animals, primary sensory cortices receive a substantial number of direct thalamic inputs from nuclei of their matched, but also from nuclei of non-matched sensory modalities. There are also direct intracortical connections among primary sensory cortices and connections with secondary sensory cortices of other modalities. In very old animals, the crossmodal connections strongly decrease in number or vanish entirely. This is likely due to a retraction of the projection neuron axonal branches rather than ongoing programmed cell death. The loss of crossmodal connections is also accompanied by changes in anatomical correlates of inhibition and excitation in the sensory thalamus and cortex. Together, the loss and restructuring of crossmodal connections during aging suggest a shift of multisensory processing from primary cortices towards other sensory brain areas in elderly individuals.

Increased cross-modal functional connectivity in cochlear implant users

Previous studies have reported increased cross-modal auditory and visual cortical activation in cochlear implant (CI) users, suggesting cross-modal reorganization of both visual and auditory cortices in CI users as a consequence of sensory deprivation and restoration. How these processes affect the functional connectivity of the auditory and visual system in CI users is however unknown. We here investigated task-induced intra-modal functional connectivity between hemispheres for both visual and auditory cortices and cross-modal functional connectivity between visual and auditory cortices using functional near infrared spectroscopy in post-lingually deaf CI users and age-matched normal hearing controls. Compared to controls, CI users exhibited decreased intra-modal functional connectivity between hemispheres and increased cross-modal functional connectivity between visual and left auditory cortices for both visual and auditory stimulus processing. Importantly, the difference between cross-modal functional connectivity for visual and for auditory stimuli correlated with speech recognition outcome in CI users. Higher cross-modal connectivity for auditory than for visual stimuli was associated with better speech recognition abilities, pointing to a new pattern of functional reorganization that is related to successful hearing restoration with a CI.

Congenital Anophthalmia and Binocular Neonatal Enucleation Differently Affect the Proteome of Primary and Secondary Visual Cortices in Mice.

In blind individuals, visually deprived occipital areas are activated by non-visual stimuli. The extent of this cross-modal activation depends on the age at onset of blindness. Cross-modal inputs have access to several anatomical pathways to reactivate deprived visual areas. Ectopic cross-modal subcortical connections have been shown in anophthalmic animals but not in animals deprived of sight at a later age. Direct and indirect cross-modal cortical connections toward visual areas could also be involved, yet the number of neurons implicated is similar between blind mice and sighted controls. Changes at the axon terminal, dendritic spine or synaptic level are therefore expected upon loss of visual inputs. Here, the proteome of V1, V2M and V2L from P0-enucleated, anophthalmic and sighted mice, sharing a common genetic background (C57BL/6J x ZRDCT/An), was investigated by 2-D DIGE and Western analyses to identify molecular adaptations to enucleation and/or anophthalmia. Few proteins were differentially expressed in enucleated or anophthalmic mice in comparison to sighted mice. The loss of sight affected three pathways: metabolism, synaptic transmission and morphogenesis. Most changes were detected in V1, followed by V2M. Overall, cross-modal adaptations could be promoted in both models of early blindness but not through the exact same molecular strategy. A lower metabolic activity observed in visual areas of blind mice suggests that even if cross-modal inputs reactivate visual areas, they could remain suboptimally processed.

Synesthetic associations and psychosensory symptoms of temporal epilepsy.

BackgroundSynesthesia manifests as unusual associative connections that may cause intriguing experiences due to various cross-modal connections, for example, a sound may be experienced as color. Several findings indicate that temporal lobe seizures or seizure-like conditions and increased excitability may influence various unusual cross-sensory links and synesthetic experiences.MethodsIn this context, the purpose of this study is to find relationships between word–color associations and psychopathological symptoms related to temporal lobe epilepsy and limbic irritability (Limbic System Checklist [LSCL-33]), symptoms of traumatic stress (Trauma Symptoms Checklist [TSC-40]), and depressive symptoms (Beck Depression Inventory [BDI-II]) in 71 participants (mean age =25.23 years) recruited from the general population. The whole sample included two subgroups according to levels of psychosensory and affective symptoms related to temporal epilepsy measured by LSCL-33.ResultsThe results in both subgroups indicate specific words correlated with the scores of psychopathological symptoms measured by LSCL-33, BDI-II, and TSC-40. Significant Spearman correlations have been predominantly found in the subgroup of participants with higher levels of LSCL-33.ConclusionThe results indicate a specific synesthetic-like mechanism in association processes that reflects psychopathological symptoms related to increased temporo-limbic excitability.

Locke, Judgment, and Figure: A Consistent Answer to the Molyneux Problem

John Locke has been famously credited with resurrecting the distinction between common and proper sensibles, better known in the Essay as primary and secondary qualities. Although some argue that Locke’s adherence to the doctrine of the common sensibles is in conflict with his empiricist sensibilities, I will show this is not likely to be the case. In order to achieve this I will argue that Locke held there to be cross-modal connections in the mind for the representational content of ideas of primary quality, through the relation of resemblance, but that these representations need to be empirically verified to be veridical. Consequently, I believe this interpretation points in a useful direction for making an epistemic distinction between objects encountered between the phenomenal and sub-phenomenal domains.