Cross-modal Recognition Research Articles

Cross-modal representation of chewing food in posterior parietal and visual cortex.

Even though the oral cavity is not visible, food chewing can be performed without damaging the tongue, oral mucosa, or other intraoral parts, with cross-modal perception of chewing possibly critical for appropriate recognition of its performance. This study was conducted to clarify the relationship of chewing food cross-modal perception with cortex activities based on examinations of the posterior parietal cortex (PPC) and visual cortex during chewing in comparison with sham chewing without food, imaginary chewing, and rest using functional near-infrared spectroscopy. Additionally, the effects of a deafferent tongue dorsum on PPC/visual cortex activities during chewing performance were examined. The results showed that chewing food increased activity in the PPC/visual cortex as compared with imaginary chewing, sham chewing without food, and rest. Nevertheless, those activities were not significantly different during imaginary chewing or sham chewing without food as compared with rest. Moreover, subjects with a deafferent tongue dorsum showed reduced PPC/visual cortex activities during chewing food performance. These findings suggest that chewing of food involves cross-modal recognition, while an oral somatosensory deficit may modulate such cross-modal activities.

First-sight recognition of touched objects shows that chicks can solve Molyneux's problem.

If a congenitally blind person learns to distinguish between a cube and a sphere by touch, would they immediately recognize these objects by sight once their vision is restored? This question, posed by Molyneux in 1688, has puzzled philosophers and scientists since then. To overcome ethical and practical difficulties in the investigation of cross-modal recognition, we studied inexperienced poultry chicks, which can be reared in darkness until the moment of a visual test with no detrimental consequences. After hatching chicks in darkness, we exposed them to either tactile smooth or tactile bumpy stimuli for 24 h. Immediately after the tactile exposure, chicks were tested in a visual recognition task, during their first experience with light. At first sight, chicks that had been exposed in the tactile modality to smooth stimuli approached the visual smooth stimulus significantly more than those exposed to the tactile bumpy stimuli. These results show that visually inexperienced chicks can solve Molyneux's problem, indicating cross-modal recognition does not require previous multimodal experience. At least in this precocial species, supra-modal brain areas appear functional already at birth. This discovery paves the way for the investigation of predisposed cross-modal cognition that does not depend on visual experience.

Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNet.

Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

Dynamic feature weakening for cross-modality person re-identification

Cross-modality person re-identification (Re-ID) can be used to perform all-weather pedestrian monitoring tasks by matching the visual features captured in both visible and infrared images. However, the existing mainstream methods mainly rely on a few high response modality-invariant features, leading to problems such as easy loss of person cues and poor algorithm robustness in the recognition process. In this paper, we propose a cross-modality recognition method with dynamic feature weakening (DFW) and flexible segmentation learning as the core mechanisms, specifically: (1) DFW is introduced into the modality-shared network to weaken the model’s over-reliance on high response regions of the image, thereby globally collecting and enriching diverse modality-invariant features. (2) A segmented learning network (SLN) with overlap penalty constraint is used for fine-grained mining of each modality-invariant feature, while avoiding redundant focus on a region and maintaining complete learning of diversity information. The experimental results show that the method in this paper greatly enriches the feature representation, and significantly improves the recognition accuracy of the model. On the SYSU-MM01 dataset, compared with the traditional modality feature learning method, the Rank1 and mAP of this method are increased by 2.56% and 4.53%, respectively.

Brain-inspired multisensory integration neural network for cross-modal recognition through spatiotemporal dynamics and deep learning

Brain-inspired multisensory integration neural network for cross-modal recognition through spatiotemporal dynamics and deep learning

Cross-modal biometric fusion intelligent traffic recognition system combined with real-time data operation

Abstract Intelligent traffic recognition system is the development direction of the future traffic system. It effectively integrates advanced information technology, data communication transmission technology, electronic sensing technology, control technology, and computer technology into the entire ground traffic management system. It establishes a real-time, accurate, and efficient integrated transportation management system that plays a role in a wide range and all directions. The aim of this article is to integrate cross-modal biometrics into an intelligent traffic recognition system combined with real-time data operations. Based on the cross-modal recognition algorithm, it can better re-identify the vehicle cross-modally by building a model. First, this article first presents a general introduction to the cross-modal recognition method. Then, the experimental analysis is conducted on the classification of vehicle images recognized by the intelligent transportation system, the complexity of vehicle logo recognition, and the recognition of vehicle images with different lights. Finally, the cross-modal recognition algorithm is introduced into the dynamic analysis of the intelligent traffic recognition system. The cross-modal traffic recognition system experiment is carried out. The experimental results show that the intraclass distribution loss function can improve the Rank 1 recognition rate and mAP value by 6–7% points on the basis of the baseline method. This shows that improving the modal invariance feature by reducing the distribution difference between different modal images of the same vehicle can effectively deal with the feature information imbalance caused by modal changes.

Deep Active Cross-Modal Visuo-Tactile Transfer Learning for Robotic Object Recognition

We propose for the first time, a novel deep active visuo-tactile cross-modal full-fledged framework for object recognition by autonomous robotic systems. Our proposed network <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xAVTNet</i> is actively trained with labelled point clouds from a vision sensor with one robot and tested with an active tactile perception strategy to recognise objects never touched before using another robot. We propose a novel visuo-tactile loss ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VTLoss</i> ) to minimise the discrepancy between the visual and tactile domains for unsupervised domain adaptation. Our framework leverages the strengths of deep neural networks for cross-modal recognition along with active perception and active learning strategies for increased efficiency by minimising redundant data collection. Our method is extensively evaluated on a real robotic system and compared against baselines and other state-of-art approaches. We demonstrate clear outperformance in recognition accuracy compared to the state-of-art visuo-tactile cross-modal recognition method.

Heterogeneous Face Interpretable Disentangled Representation for Joint Face Recognition and Synthesis.

Heterogeneous faces are acquired with different sensors, which are closer to real-world scenarios and play an important role in the biometric security field. However, heterogeneous face analysis is still a challenging problem due to the large discrepancy between different modalities. Recent works either focus on designing a novel loss function or network architecture to directly extract modality-invariant features or synthesizing the same modality faces initially to decrease the modality gap. Yet, the former always lacks explicit interpretability, and the latter strategy inherently brings in synthesis bias. In this article, we explore to learn the plain interpretable representation for complex heterogeneous faces and simultaneously perform face recognition and synthesis tasks. We propose the heterogeneous face interpretable disentangled representation (HFIDR) that could explicitly interpret dimensions of face representation rather than simple mapping. Benefited from the interpretable structure, we further could extract latent identity information for cross-modality recognition and convert the modality factor to synthesize cross-modality faces. Moreover, we propose a multimodality heterogeneous face interpretable disentangled representation (M-HFIDR) to extend the basic approach suitable for the multimodality face recognition and synthesis. To evaluate the ability of generalization, we construct a novel large-scale face sketch data set. Experimental results on multiple heterogeneous face databases demonstrate the effectiveness of the proposed method.

A Discriminative Vectorial Framework for Multi-Modal Feature Representation

Due to the rapid advancements of sensory and computing technology, multi-modal data sources that represent the same pattern or phenomenon have attracted growing attention. As a result, finding means to explore useful information from these multi-modal data sources has quickly become a necessity. In this paper, a discriminative vectorial framework is proposed for multi-modal feature representation in knowledge discovery by employing multi-modal hashing (MH) and discriminative correlation maximization (DCM) analysis. Specifically, the proposed framework is capable of minimizing the semantic similarity among different modalities by MH and exacting intrinsic discriminative representations across multiple data sources by DCM analysis jointly, enabling a novel vectorial framework of multi-modal feature representation. Moreover, the proposed feature representation strategy is analyzed and further optimized based on canonical and non-canonical cases, respectively. Consequently, the generated feature representation leads to effective utilization of the input data sources of high quality, producing improved, sometimes quite impressive, results in various applications. The effectiveness and generality of the proposed framework are demonstrated by utilizing classical features and deep neural network (DNN) based features with applications to image and multimedia analysis and recognition tasks, including data visualization, face recognition, object recognition; cross-modal (text-image) recognition and audio emotion recognition. Experimental results show that the proposed solutions are superior to state-of-the-art statistical machine learning (SML) and DNN algorithms.

Dual adaptive alignment and partitioning network for visible and infrared cross-modality person re-identification

Visible and infrared person re-identification (VI-ReID) describes the task of matching the images of a person, captured by visible-light and infrared cameras; this is a particular challenge in night time surveillance applications. Existing cross-modality recognition studies have been conducted mainly with a focus on learning the global and shareable feature representation of pedestrians to handle cross-modality discrepancies. However, the global features of pedestrian images cannot solve the unaligned image pairs efficiently, particularly when encountering the human appearance or posture misalignment caused by inaccurate pedestrian detection boxes. To mitigate the impact of these problems, we propose an end-to-end dual alignment and partitioning network to simultaneously learn global and local modal invariant features of pedestrians. First, we use two adaptive spatial transform modules to align the visible and infrared input images. Subsequently, the aligned image is divided horizontally, and the features of each local block are extracted. Then, we fuse these local features with global features. To alleviate the differences between heterogeneous modals and learn the common feature representation of heterogeneous modals, we map the features of heterogeneous modes into the same feature embedding space. Finally, we use the combination of identity loss and weighted regularized TriHard loss to improve the recognition accuracy. Extensive experimental results on two cross-modality datasets, RegDB and SYSU-MM01, demonstrate the superiority of the proposed method over other existing state-of-the-art methods.

Bridging ears and eyes when learning spoken words: On the effects of bilingual experience at 30months.

From the very first moments of their lives, infants selectively attend to the visible orofacial movements of their social partners and apply their exquisite speech perception skills to the service of lexical learning. Here we explore how early bilingual experience modulates children's ability to use visible speech as they form new lexical representations. Using a cross-modal word-learning task, bilingual children aged 30months were tested on their ability to learn new lexical mappings in either the auditory or the visual modality. Lexical recognition was assessed either in the same modality as the one used at learning ('same modality' condition: auditory test after auditory learning, visual test after visual learning) or in the other modality ('cross-modality' condition: visual test after auditory learning, auditory test after visual learning). The results revealed that like their monolingual peers, bilingual children successfully learn new words in either the auditory or the visual modality and show cross-modal recognition of words following auditory learning. Interestingly, as opposed to monolinguals, they also demonstrate cross-modal recognition of words upon visual learning. Collectively, these findings indicate a bilingual edge in visual word learning, expressed in the capacity to form a recoverable cross-modal representation of visually learned words.

These bees have “seen” that before

Animal Cognition Humans excel at mental imagery, and we can transfer those images across senses. For example, an object out of view, but for which we have a mental image, can still be recognized by touch. Such cross-modal recognition is highly adaptive and has been recently identified in other mammals, but whether it is widespread has been debated. Solvi et al. tested for this behavior in bumble bees, which are increasingly recognized as having some relatively advanced cognitive skills (see the Perspective by von der Emde and Burt de Perera). They found that the bees could identify objects by shape in the dark if they had seen, but not touched, them in the light, and vice versa, demonstrating a clear ability to transmit recognition across senses. Science , this issue p. [910][1]; see also p. [850][2] [1]: /lookup/doi/10.1126/science.aay8064 [2]: /lookup/doi/10.1126/science.aba8519

Bumble bees display cross-modal object recognition between visual and tactile senses.

Many animals can associate object shapes with incentives. However, such behavior is possible without storing images of shapes in memory that are accessible to more than one sensory modality. One way to explore whether there are modality-independent internal representations of object shapes is to investigate cross-modal recognition-experiencing an object in one sensory modality and later recognizing it in another. We show that bumble bees trained to discriminate two differently shaped objects (cubes and spheres) using only touch (in darkness) or vision (in light, but barred from touching the objects) could subsequently discriminate those same objects using only the other sensory information. Our experiments demonstrate that bumble bees possess the ability to integrate sensory information in a way that requires modality-independent internal representations.

Horses Categorize Human Emotions Cross-Modally Based on Facial Expression and Non-Verbal Vocalizations.

Simple SummaryRecently, an increasing number of studies have investigated the expression and perception of emotions by non-human animals. In particular, it is of interest to determine whether animals can link emotion stimuli of different modalities (e.g., visual and oral) based on the emotions that are expressed (i.e., to recognize emotions cross-modally). For domestic species that share a close relationship with humans, we might even wonder whether this ability extends to human emotions. Here, we investigated whether domestic horses recognize human emotions cross-modally. We simultaneously presented two animated pictures of human facial expressions, one typical of joy and the other of anger; simultaneously, a speaker played a human non-verbal vocalization expressing joy or anger. Horses looked at the picture that did not match the emotion of the vocalization more (probably because they were intrigued by the paradoxical combination). Moreover, their behavior and heart rate differed depending on the vocalization: they reacted more negatively to the anger vocalization and more positively to the joy vocalization. These results suggest that horses can match visual and vocal cues for the same emotion and can perceive the emotional valence of human non-verbal vocalizations.Over the last few years, an increasing number of studies have aimed to gain more insight into the field of animal emotions. In particular, it is of interest to determine whether animals can cross-modally categorize the emotions of others. For domestic animals that share a close relationship with humans, we might wonder whether this cross-modal recognition of emotions extends to humans, as well. In this study, we tested whether horses could recognize human emotions and attribute the emotional valence of visual (facial expression) and vocal (non-verbal vocalization) stimuli to the same perceptual category. Two animated pictures of different facial expressions (anger and joy) were simultaneously presented to the horses, while a speaker played an emotional human non-verbal vocalization matching one of the two facial expressions. Horses looked at the picture that was incongruent with the vocalization more, probably because they were intrigued by the paradoxical combination. Moreover, horses reacted in accordance with the valence of the vocalization, both behaviorally and physiologically (heart rate). These results show that horses can cross-modally recognize human emotions and react emotionally to the emotional states of humans, assessed by non-verbal vocalizations.

Bi-Directional Center-Constrained Top-Ranking for Visible Thermal Person Re-Identification

Visible thermal person re-identification (VT-REID) is a task of matching person images captured by thermal and visible cameras, which is an extremely important issue in night-time surveillance applications. Existing cross-modality recognition works mainly focus on learning sharable feature representations to handle the cross-modality discrepancies. However, apart from the cross-modality discrepancy caused by different camera spectrums, VT-REID also suffers from large cross-modality and intra-modality variations caused by different camera environments and human poses, and so on. In this paper, we propose a dual-path network with a novel bi-directional dual-constrained top-ranking (BDTR) loss to learn discriminative feature representations. It is featured in two aspects: 1) end-to-end learning without extra metric learning step and 2) the dual-constraint simultaneously handles the cross-modality and intra-modality variations to ensure the feature discriminability. Meanwhile, a bi-directional center-constrained top-ranking (eBDTR) is proposed to incorporate the previous two constraints into a single formula, which preserves the properties to handle both cross-modality and intra-modality variations. The extensive experiments on two cross-modality re-ID datasets demonstrate the superiority of the proposed method compared to the state-of-the-arts.

Is Locke’s answer to Molyneux’s question inconsistent? Cross-modal recognition and the sight–recognition error

AbstractMolyneux’s question asks whether someone born blind, who could distinguish cubes from spheres using his tactile sensation, could recognize those objects if he received his sight. Locke says no: the newly sighted person would fail to point to the cube and call it a cube. Locke never provided a complete explanation for his negative response, and there are concerns of inconsistency with other important aspects of his theory of ideas. These charges of inconsistency rest upon an unrecognized and unfounded assumption that seeing entails recognition. Locke’s negative answer to Molyneux’s question is consistent with his other philosophical commitments.

Visuo-haptic transfer for object recognition in children with periventricular leukomalacia and bilateral cerebral palsy

ABSTRACTObject recognition is a complex adaptive process that can be impaired in children with neurodevelopmental disabilities. Recently, we found a significant effect of time on the development of unimodal and crossmodal recognition skills for common objects in typical children and this was a starting point for the study of visuo-haptic object recognition skills in impaired populations. In this study, we investigated unimodal visual information, unimodal haptic information and visuo-haptic information transfer in 30 children, from 4.0 to 10.11 years of age, with bilateral Periventricular Leukomalacia (PVL) and bilateral cerebral palsy. Results were matched with those of 116 controls. Participants were tested using a clinical protocol, adopted in the previous study, involving visual exploration of black-and-white photographs of common objects, haptic exploration of real objects and visuo-haptic transfer of these two types of information. Results show that in the PVL group as in controls, there is an age-dependent development of object recognition abilities for visual, haptic and visuo-haptic modalities, even if PVL children perform worse in all the three conditions, in comparison with the typical group. Furthermore, PVL children have a specific deficit both in visual and haptic information processing, that improves with age, probably thanks to everyday experience, but the visual modality shows a better and more rapid maturation, remaining more salient compared to the haptic one. However, multisensory processes partially facilitate recognition of common objects also in PVL children and this finding could be useful for planning early intervention in children with brain lesion.

Maternal Immune Activation during Pregnancy Alters the Behavior Profile of Female Offspring of Sprague Dawley Rats.

Sex differences are documented in psychiatric and neurological disorders, yet most preclinical animal research has been conducted in males only. There is a need to better understand of the nature of sex differences in brain disease in order to meet the needs of psychiatric patients. We present the behavior profile of adult female offspring produced using a maternal immune activation (MIA) model where pregnant rats receive an immune stimulant and the offspring typically show various abnormalities consistent with psychiatric illnesses such as schizophrenia and autism. The results in female offspring were compared to a previously published cohort of their male siblings (Lins et al., 2018). We examined prepulse inhibition (PPI), sociability, MK-801-induced locomotor activity, crossmodal object recognition (CMOR), and oddity discrimination; behaviors relevant to the positive, negative, and cognitive symptoms of schizophrenia. No between-treatment differences in PPI or locomotor activity were noted. Tactile memory was observed in the control and treated female offspring, visual recognition memory was deficient in the polyinosinic:polycytidylic acid (polyI:C) offspring only, and both groups lacked crossmodal recognition. PolyI:C offspring were impaired in oddity preference and had reduced preference for a stranger conspecific in a sociability assay. Systemic maternal CXCL1, IL-6, and TNF-a levels 3 h after polyI:C treatment were determined, but no relationship was found between these cytokines and the behavior seen in the adult female offspring. Overall, female offspring of polyI:C-treated dams display an array of behavior abnormalities relevant to psychiatric illnesses such as schizophrenia similar to those previously reported in male rats.

Impaired discrimination with intact crossmodal association in aged rats: A dissociation of perirhinal cortical-dependent behaviors.

The perirhinal cortex (PRC) supports associative memory and perception, and PRC dysfunction impairs animals' abilities to associate stimulus features across sensory modalities. PRC damage also leads to deficits in discriminating between stimuli that share features. Although PRC-dependent stimulus discrimination has been shown to be impaired with advanced age, data regarding the abilities of older adults and other animals to form PRC-dependent associations have been equivocal. Moreover, the extent to which similar neural computations within the PRC support associative memory versus discrimination abilities have not been directly examined. In the current study, young and aged rats were cross-characterized on two PRC-dependent crossmodal object recognition (CMOR) tasks to test associative memory, and a LEGO object discrimination task. In the CMOR tasks, rats were familiarized with an object with access to tactile input and then tested for recognition with visual input only. The relative exploration time of novel versus familiar objects indicated that aged rats showed preference for the novel over familiar object with and without an epoch of multimodal preexposure to the familiar object prior to the testing session. Furthermore, crossmodal recognition performance between young and aged rats was not significantly different. In contrast, for the LEGO object discrimination task, aged rats were impaired relative to young rats. Notably, aged rats that performed poorly on the LEGO object discrimination task had better performance on the CMOR tasks. The dissociation of discrimination and association abilities with age suggests that these behaviors rely on distinct neural computations within PRC-medial temporal lobe circuit. (PsycINFO Database Record

Effects of modality and repetition in a continuous recognition memory task: Repetition has no effect on auditory recognition memory

Previous research has shown that auditory recognition memory is poorer compared to visual and cross-modal (visual and auditory) recognition memory. The effect of repetition on memory has been robust in showing improved performance. It is not clear, however, how auditory recognition memory compares to visual and cross-modal recognition memory following repetition. Participants performed a recognition memory task, making old/new discriminations to new stimuli, stimuli repeated for the first time after 4–7 intervening items (R1), or repeated for the second time after 36–39 intervening items (R2). Depending on the condition, participants were either exposed to visual stimuli (2D line drawings), auditory stimuli (spoken words), or cross-modal stimuli (pairs of images and associated spoken words). Results showed that unlike participants in the visual and cross-modal conditions, participants in the auditory recognition did not show improvements in performance on R2 trials compared to R1 trials. These findings have implications for pedagogical techniques in education, as well as for interventions and exercises aimed at boosting memory performance.