Cross-modal Recognition Research Articles

Numerical assessment in the wild: insights from social carnivores.

Playback experiments have proved to be a useful tool to investigate the extent to which wild animals understand numerical concepts and the factors that play into their decisions to respond to different numbers of vocalizing conspecifics. In particular, playback experiments have broadened our understanding of the cognitive abilities of historically understudied species that are challenging to test in the traditional laboratory, such as members of the Order Carnivora. Additionally, playback experiments allow us to assess the importance of numerical information versus other ecologically important variables when animals are making adaptive decisions in their natural habitats. Here, we begin by reviewing what we know about quantity discrimination in carnivores from studies conducted in captivity. We then review a series of playback experiments conducted with wild social carnivores, including African lions, spotted hyenas and wolves, which demonstrate that these animals can assess the number of conspecifics calling and respond based on numerical advantage. We discuss how the wild studies complement those conducted in captivity and allow us to gain insights into why wild animals may not always respond based solely on differences in quantity. We then consider the key roles that individual discrimination and cross-modal recognition play in the ability of animals to assess the number of conspecifics vocalizing nearby. Finally, we explore new directions for future research in this area, highlighting in particular the need for further work on the cognitive basis of numerical assessment skills and experimental paradigms that can be effective in both captive and wild settings.This article is part of a discussion meeting issue 'The origins of numerical abilities'.

Learning Spoken Words via the Ears and Eyes: Evidence from 30-Month-Old Children.

From the very first moments of their lives, infants are able to link specific movements of the visual articulators to auditory speech signals. However, recent evidence indicates that infants focus primarily on auditory speech signals when learning new words. Here, we ask whether 30-month-old children are able to learn new words based solely on visible speech information, and whether information from both auditory and visual modalities is available after learning in only one modality. To test this, children were taught new lexical mappings. One group of children experienced the words in the auditory modality (i.e., acoustic form of the word with no accompanying face). Another group experienced the words in the visual modality (seeing a silent talking face). Lexical recognition was tested in either the learning modality or in the other modality. Results revealed successful word learning in either modality. Results further showed cross-modal recognition following an auditory-only, but not a visual-only, experience of the words. Together, these findings suggest that visible speech becomes increasingly informative for the purpose of lexical learning, but that an auditory-only experience evokes a cross-modal representation of the words.

Specific Patterns of Emotion Recognition from Faces in Children with ASD: Results of a Cross-Modal Matching Paradigm.

Children with ASD show emotion recognition difficulties, as part of their social communication deficits. We examined facial emotion recognition (FER) in intellectually disabled children with ASD and in younger typically developing (TD) controls, matched on mental age. Our emotion-matching paradigm employed three different modalities: facial, vocal and verbal. Results confirmed overall FER deficits in ASD. Compared to the TD group, children with ASD had the poorest performance in recognizing surprise and anger in comparison to happiness and sadness, and struggled with face-face matching, compared to voice-face and word-face combinations. Performance in the voice-face cross-modal recognition task was related to adaptive communication. These findings highlight the specific face processing deficit, and the relative merit of cross-modal integration in children with ASD.

Simultaneous Semi-Coupled Dictionary Learning for Matching in Canonical Space.

Cross-modal recognition and matching with privileged information are important challenging problems in the field of computer vision. The cross-modal scenario deals with matching across different modalities and needs to take care of the large variations present across and within each modality. The privileged information scenario deals with the situation that all the information available during training may not be available during the testing stage, and hence, algorithms need to leverage the extra information from the training stage itself. We show that for multi-modal data, either one of the above situations may arise if one modality is absent during testing. Here, we propose a novel framework, which can handle both these scenarios seamlessly with applications to matching multi-modal data. The proposed approach jointly uses data from the two modalities to build a canonical representation, which encompasses information from both the modalities. We explore four different types of canonical representations for different types of data. The algorithm computes dictionaries and canonical representation for data from both the modalities, such that the transformed sparse coefficients of both the modalities are equal to that of the canonical representation. The sparse coefficients are finally matched using Mahalanobis metric. Extensive experiments on different data sets, involving RGBD, text-image, and audio-image data, show the effectiveness of the proposed framework.

Development of visuo-haptic transfer for object recognition in typical preschool and school-aged children

ABSTRACTObject recognition is a long and complex adaptive process and its full maturation requires combination of many different sensory experiences as well as cognitive abilities to manipulate previous experiences in order to develop new percepts and subsequently to learn from the environment. It is well recognized that the transfer of visual and haptic information facilitates object recognition in adults, but less is known about development of this ability. In this study, we explored the developmental course of object recognition capacity in children using unimodal visual information, unimodal haptic information, and visuo-haptic information transfer in children from 4 years to 10 years and 11 months of age. Participants were tested through a clinical protocol, involving visual exploration of black-and-white photographs of common objects, haptic exploration of real objects, and visuo-haptic transfer of these two types of information. Results show an age-dependent development of object recognition abilities for visual, haptic, and visuo-haptic modalities. A significant effect of time on development of unimodal and crossmodal recognition skills was found. Moreover, our data suggest that multisensory processes for common object recognition are active at 4 years of age. They facilitate recognition of common objects, and, although not fully mature, are significant in adaptive behavior from the first years of age. The study of typical development of visuo-haptic processes in childhood is a starting point for future studies regarding object recognition in impaired populations.

Cross-modal recognition of familiar conspecifics in goats.

When identifying other individuals, animals may match current cues with stored information about that individual from the same sensory modality. Animals may also be able to combine current information with previously acquired information from other sensory modalities, indicating that they possess complex cognitive templates of individuals that are independent of modality. We investigated whether goats (Capra hircus) possess cross-modal representations (auditory–visual) of conspecifics. We presented subjects with recorded conspecific calls broadcast equidistant between two individuals, one of which was the caller. We found that, when presented with a stablemate and another herd member, goats looked towards the caller sooner and for longer than the non-caller, regardless of caller identity. By contrast, when choosing between two herd members, other than their stablemate, goats did not show a preference to look towards the caller. Goats show cross-modal recognition of close social partners, but not of less familiar herd members. Goats may employ inferential reasoning when identifying conspecifics, potentially facilitating individual identification based on incomplete information. Understanding the prevalence of cross-modal recognition and the degree to which different sensory modalities are integrated provides insight into how animals learn about other individuals, and the evolution of animal communication.

カラスのコミュニケーション:聞く・見る・触れる

Large-billed crows, Corvus macrorhynchos, are one of familiar resident birds in Japan. They are originally jungle/forest-dwelling birds but now broaden their habitats to urban 'concrete jungle' environments such as Tokyo and Sapporo. Despite the close and overlapping range of the habitat between humans and crows, the socio-behavioural function of crows and its psychological and neural underpinnings have been poorly understood. Here, I review the recent studies in animal psychology on inter-individual communication for the understanding of individual-based social ecology of large-billed crows. Particular focuses are made on audio-vocal, visual, and tactile communication. First, vocal exchange with a contact call (ka call) and the acoustic individuality of ka calls as an identity signal are outlined. Second topic for visual communication is on audio-visual cross-modal recognition of familiar individuals and the involvement of the individual recognition in the formation and maintenance of dominance relationship. Finally, nonreciprocal but unidirectional allopreening from dominants to subordinates in juvenile males is shown and its possible social function are discussed. To further understand the communication of crows from comparative perspectives, these findings of social behaviour and its function should be integrated with physiological research on the... onto the life-history axis of this species which is characterized by drastic change of social structures between juvenile and pair-bond stages.

Thermal-to-visible face recognition using partial least squares.

Although visible face recognition has been an active area of research for several decades, cross-modal face recognition has only been explored by the biometrics community relatively recently. Thermal-to-visible face recognition is one of the most difficult cross-modal face recognition challenges, because of the difference in phenomenology between the thermal and visible imaging modalities. We address the cross-modal recognition problem using a partial least squares (PLS) regression-based approach consisting of preprocessing, feature extraction, and PLS model building. The preprocessing and feature extraction stages are designed to reduce the modality gap between the thermal and visible facial signatures, and facilitate the subsequent one-vs-all PLS-based model building. We incorporate multi-modal information into the PLS model building stage to enhance cross-modal recognition. The performance of the proposed recognition algorithm is evaluated on three challenging datasets containing visible and thermal imagery acquired under different experimental scenarios: time-lapse, physical tasks, mental tasks, and subject-to-camera range. These scenarios represent difficult challenges relevant to real-world applications. We demonstrate that the proposed method performs robustly for the examined scenarios.

Behavioral alterations in rat offspring following maternal immune activation and ELR-CXC chemokine receptor antagonism during pregnancy: implications for neurodevelopmental psychiatric disorders.

Research suggests that maternal immune activation (MIA) during pregnancy increases the risk of neurodevelopmental disorders including schizophrenia and autism in the offspring. Current theories suggest that inflammatory mediators including cytokines and chemokines may underlie the increased risk of these disorders in humans. For example, elevated maternal interleukin-8 (IL-8) during pregnancy is associated with increased risk of schizophrenia in the offspring. Given this association, the present experiments examined ELR-CXC chemokines CXCL1 and CXCL2, rodent homologues of human IL-8, and activation of their receptors (CXCR1 and CXCR2) in an established rodent model of MIA. Pregnant Long Evans rats were treated with the viral mimetic polyinosinic-polycytidylic acid (polyI:C; 4 mg/kg, i.v.) on gestational day 15. Protein analysis using multiplex assays and ELISA showed that polyI:C significantly increased maternal serum concentrations of interleukin-1β, tumor necrosis factor, and CXCL1 3h after administration. Subsequent experiments tested the role of elevated maternal CXCL1 on behavior of the offspring by administering a CXCR1/CXCR2 antagonist (G31P; 500 μg/kg, i.p.; 1h before, 48 and 96 h after polyI:C treatment). The male offspring of dams treated with polyI:C demonstrated subtle impairments in prepulse inhibition (PPI), impaired associative and crossmodal recognition memory, and altered behavioral flexibility in an operant test battery. While G31P did not completely reverse the behavioral impairments caused by polyI:C, it enhanced PPI during adolescence and strategy set-shifting and reversal learning during young adulthood. These results suggest that while polyI:C treatment significantly increases maternal CXCL1, elevations of this chemokine are not solely responsible for the effects of polyI:C on the behavior of the offspring.

How the human brain exchanges information across sensory modalities to recognize other people.

Recognizing the identity of other individuals across different sensory modalities is critical for successful social interaction. In the human brain, face- and voice-sensitive areas are separate, but structurally connected. What kind of information is exchanged between these specialized areas during cross-modal recognition of other individuals is currently unclear. For faces, specific areas are sensitive to identity and to physical properties. It is an open question whether voices activate representations of face identity or physical facial properties in these areas. To address this question, we used functional magnetic resonance imaging in humans and a voice-face priming design. In this design, familiar voices were followed by morphed faces that matched or mismatched with respect to identity or physical properties. The results showed that responses in face-sensitive regions were modulated when face identity or physical properties did not match to the preceding voice. The strength of this mismatch signal depended on the level of certainty the participant had about the voice identity. This suggests that both identity and physical property information was provided by the voice to face areas. The activity and connectivity profiles differed between face-sensitive areas: (i) the occipital face area seemed to receive information about both physical properties and identity, (ii) the fusiform face area seemed to receive identity, and (iii) the anterior temporal lobe seemed to receive predominantly identity information from the voice. We interpret these results within a prediction coding scheme in which both identity and physical property information is used across sensory modalities to recognize individuals.

Dysgranular retrosplenial cortex lesions in rats disrupt cross-modal object recognition

The retrosplenial cortex supports navigation, with one role thought to be the integration of different spatial cue types. This hypothesis was extended by examining the integration of nonspatial cues. Rats with lesions in either the dysgranular subregion of retrosplenial cortex (area 30) or lesions in both the granular and dysgranular subregions (areas 29 and 30) were tested on cross-modal object recognition (Experiment 1). In these tests, rats used different sensory modalities when exploring and subsequently recognizing the same test objects. The objects were first presented either in the dark, i.e., giving tactile and olfactory cues, or in the light behind a clear Perspex barrier, i.e., giving visual cues. Animals were then tested with either constant combinations of sample and test conditions (light to light, dark to dark), or changed “cross-modal” combinations (light to dark, dark to light). In Experiment 2, visual object recognition was tested without Perspex barriers, but using objects that could not be distinguished in the dark. The dysgranular retrosplenial cortex lesions selectively impaired cross-modal recognition when cue conditions switched from dark to light between initial sampling and subsequent object recognition, but no impairment was seen when the cue conditions remained constant, whether dark or light. The combined (areas 29 and 30) lesioned rats also failed the dark to light cross-modal problem but this impairment was less selective. The present findings suggest a role for the dysgranular retrosplenial cortex in mediating the integration of information across multiple cue types, a role that potentially applies to both spatial and nonspatial domains.

Alternation of clear-cut colour patterns inCorvuscrow evolution accords with learning-dependent social selection against unusual-looking conspecifics

Alternation of clear-cut colour patterns in<i>Corvus</i>crow evolution accords with learning-dependent social selection against unusual-looking conspecifics

Assessing crossmodal matching of abstract auditory and visual stimuli in posterior superior temporal sulcus with MEG

Associating crossmodal auditory and visual stimuli is an important component of perception, with the posterior superior temporal sulcus (pSTS) hypothesized to support this. However, recent evidence has argued that the pSTS serves to associate two stimuli irrespective of modality. To examine the contribution of pSTS to crossmodal recognition, participants (N=13) learned 12 abstract, non-linguistic pairs of stimuli over 3weeks. These paired associates comprised four types: auditory–visual (AV), auditory–auditory (AA), visual–auditory (VA), and visual–visual (VV). At week four, participants were scanned using magnetoencephalography (MEG) while performing a correct/incorrect judgment on pairs of items. Using an implementation of synthetic aperture magnetometry that computes real statistics across trials (SAMspm), we directly contrasted crossmodal (AV and VA) with unimodal (AA and VV) pairs from stimulus-onset to 2s in theta (4–8Hz), alpha (9–15Hz), beta (16–30Hz), and gamma (31–50Hz) frequencies. We found pSTS showed greater desynchronization in the beta frequency for crossmodal compared with unimodal trials, suggesting greater activity during the crossmodal pairs, which was not influenced by congruency of the paired stimuli. Using a sliding window SAM analysis, we found the timing of this difference began in a window from 250 to 750ms after stimulus-onset. Further, when we directly contrasted all sub-types of paired associates from stimulus-onset to 2s, we found that pSTS seemed to respond to dynamic, auditory stimuli, rather than crossmodal stimuli per se. These findings support an early role for pSTS in the processing of dynamic, auditory stimuli, and do not support claims that pSTS is responsible for associating two stimuli irrespective of their modality.

Cross-modal individual recognition in domestic horses ( Equus caballus ) extends to familiar humans

It has recently been shown that some non-human animals can cross-modally recognize members of their own taxon. What is unclear is just how plastic this recognition system can be. In this study, we investigate whether an animal, the domestic horse, is capable of spontaneous cross-modal recognition of individuals from a morphologically very different species. We also provide the first insights into how cross-modal identity information is processed by examining whether there are hemispheric biases in this important social skill. In our preferential looking paradigm, subjects were presented with two people and playbacks of their voices to determine whether they were able to match the voice with the person. When presented with familiar handlers subjects could match the specific familiar person with the correct familiar voice. Horses were significantly better at performing the matching task when the congruent person was standing on their right, indicating marked hemispheric specialization (left hemisphere bias) in this ability. These results are the first to demonstrate that cross-modal recognition in animals can extend to individuals from phylogenetically very distant species. They also indicate that processes governed by the left hemisphere are central to the cross-modal matching of visual and auditory information from familiar individuals in a naturalistic setting.

Cross-modal recognition of human individuals in domestic horses (Equus caballus)

This study has shown that domestic horses are capable of cross-modal recognition of familiar humans. It was demonstrated that horses are able to discriminate between the voices of a familiar and an unfamiliar human without seeing or smelling them at the same moment. Conversely, they were able to discriminate the same persons when only exposed to their visual and olfactory cues, without being stimulated by their voices. A cross-modal expectancy violation setup was employed; subjects were exposed both to trials with incongruent auditory and visual/olfactory identity cues and trials with congruent cues. It was found that subjects responded more quickly, longer and more often in incongruent trials, exhibiting heightened interest in unmatched cues of identity. This suggests that the equine brain is able to integrate multisensory identity cues from a familiar human into a person representation that allows the brain, when deprived of one or two senses, to maintain recognition of this person.

Crows cross-modally recognize group members but not non-group members

Recognizing other individuals by integrating different sensory modalities is a crucial ability of social animals, including humans. Although cross-modal individual recognition has been demonstrated in mammals, the extent of its use by birds remains unknown. Herein, we report the first evidence of cross-modal recognition of group members by a highly social bird, the large-billed crow (Corvus macrorhynchos). A cross-modal expectancy violation paradigm was used to test whether crows were sensitive to identity congruence between visual presentation of a group member and the subsequent playback of a contact call. Crows looked more rapidly and for a longer duration when the visual and auditory stimuli were incongruent than when congruent. Moreover, these responses were not observed with non-group member stimuli. These results indicate that crows spontaneously associate visual and auditory information of group members but not of non-group members, which is a demonstration of cross-modal audiovisual recognition of group members in birds.

EEG-power and -coherence changes in a unimodal and a crossmodal working memory task with visual and kinesthetic stimuli

We investigated EEG-power and EEG-coherence changes in a unimodal and a crossmodal matching-to-sample working memory task with either visual or kinesthetic stimuli. Angle-shaped trajectories were used as stimuli presented either as a moving dot on a screen or as a passive movement of a haptic device. Effects were evaluated during the different phases of encoding, maintenance, and recognition. Alpha power was modulated during encoding by the stimulus modality, and in crossmodal conditions during encoding and maintenance by the expected modality of the upcoming test stimulus. These power modulations were observed over modality-specific cortex regions. Systematic changes of coherence for crossmodal compared to unimodal tasks were not observed during encoding and maintenance but only during recognition. There, coherence in the theta-band increased between electrode sites over left central and occipital cortex areas in the crossmodal compared to the unimodal conditions. The results underline the importance of modality-specific representations and processes in unimodal and crossmodal working memory tasks. Crossmodal recognition of visually and kinesthetically presented object features seems to be related to a direct interaction of somatosensory/motor and visual cortex regions by means of long-range synchronization in the theta-band and such interactions seem to take place at the beginning of the recognition phase, i.e. when crossmodal transfer is actually necessary.

Voice processing in dementia: a neuropsychological and neuroanatomical analysis

Voice processing in neurodegenerative disease is poorly understood. Here we undertook a systematic investigation of voice processing in a cohort of patients with clinical diagnoses representing two canonical dementia syndromes: temporal variant frontotemporal lobar degeneration (n = 14) and Alzheimer’s disease (n = 22). Patient performance was compared with a healthy matched control group (n = 35). All subjects had a comprehensive neuropsychological assessment including measures of voice perception (vocal size, gender, speaker discrimination) and voice recognition (familiarity, identification, naming and cross-modal matching) and equivalent measures of face and name processing. Neuroanatomical associations of voice processing performance were assessed using voxel-based morphometry. Both disease groups showed deficits on all aspects of voice recognition and impairment was more severe in the temporal variant frontotemporal lobar degeneration group than the Alzheimer’s disease group. Face and name recognition were also impaired in both disease groups and name recognition was significantly more impaired than other modalities in the temporal variant frontotemporal lobar degeneration group. The Alzheimer’s disease group showed additional deficits of vocal gender perception and voice discrimination. The neuroanatomical analysis across both disease groups revealed common grey matter associations of familiarity, identification and cross-modal recognition in all modalities in the right temporal pole and anterior fusiform gyrus; while in the Alzheimer’s disease group, voice discrimination was associated with grey matter in the right inferior parietal lobe. The findings suggest that impairments of voice recognition are significant in both these canonical dementia syndromes but particularly severe in temporal variant frontotemporal lobar degeneration, whereas impairments of voice perception may show relative specificity for Alzheimer’s disease. The right anterior temporal lobe is likely to have a critical role in the recognition of voices and other modalities of person knowledge.

Rhesus Monkeys See Who They Hear: Spontaneous Cross-Modal Memory for Familiar Conspecifics

Rhesus monkeys gather much of their knowledge of the social world through visual input and may preferentially represent this knowledge in the visual modality. Recognition of familiar faces is clearly advantageous, and the flexibility and utility of primate social memory would be greatly enhanced if visual memories could be accessed cross-modally either by visual or auditory stimulation. Such cross-modal access to visual memory would facilitate flexible retrieval of the knowledge necessary for adaptive social behavior. We tested whether rhesus monkeys have cross-modal access to visual memory for familiar conspecifics using a delayed matching-to-sample procedure. Monkeys learned visual matching of video clips of familiar individuals to photographs of those individuals, and generalized performance to novel videos. In crossmodal probe trials, coo-calls were played during the memory interval. The calls were either from the monkey just seen in the sample video clip or from a different familiar monkey. Even though the monkeys were trained exclusively in visual matching, the calls influenced choice by causing an increase in the proportion of errors to the picture of the monkey whose voice was heard on incongruent trials. This result demonstrates spontaneous cross-modal recognition. It also shows that viewing videos of familiar monkeys activates naturally formed memories of real monkeys, validating the use of video stimuli in studies of social cognition in monkeys.

Integration of faces and voices, but not faces and names, in person recognition

Recent studies on cross-modal recognition suggest that face and voice information are linked for the purpose of person identification. We tested whether congruent associations between familiarized faces and voices facilitated subsequent person recognition relative to incongruent associations. Furthermore, we investigated whether congruent face and name associations would similarly benefit person identification relative to incongruent face and name associations. Participants were familiarized with a set of talking video-images of actors, their names, and their voices. They were then tested on their recognition of either the face, voice, or name of each actor from bimodal stimuli which were either congruent or novel (incongruent) associations between the familiarized face and voice or face and name. We found that response times to familiarity decisions based on congruent face and voice stimuli were facilitated relative to incongruent associations. In contrast, we failed to find a benefit for congruent face and name pairs. Our findings suggest that faces and voices, but not faces and names, are integrated in memory for the purpose of person recognition. These findings have important implications for current models of face perception and support growing evidence for multisensory effects in face perception areas of the brain for the purpose of person recognition.