Cross-modal Recognition Research Articles

The functional anatomy of visual-tactile integration in man: a study using positron emission tomography

The integration of neural signals from different sensory modalities is a prerequisite for many cognitive and behavioural functions. In this study, we have mapped the functional anatomy of the integration of sensory signals across the tactile and visual modalities. Using the PET radiotracer H 2 15O, regional cerebral blood flow (rCBF) changes were measured in eight normal volunteers performing crossmodal recognition of simultaneously presented visual and tactile stimuli using a modified version of the ‘arc-circle test’. Whilst intramodal matching within the visual modality led to relative rCBF increases in the visual association cortex, crossmodal matching (visual–tactile), when compared to intramodal matching, was accompanied by relative rCBF increases in the anterior cingulate cortex, inferior parietal lobules, the left dorsolateral prefrontal cortex (DLPFC) and the left claustrum/insular cortex. The pattern of brain activation is congruent with areas of heteromodal and supramodal cortex and indicates that activation of multimodal areas is required to solve the crossmodal problem.

Children's haptic and cross-modal recognition with familiar and unfamiliar objects.

Five-year-old children explored multidimensional objects either haptically or visually and then were tested for recognition with target and distractor items in either the same or the alternative modality. In Experiments 1 and 2, haptic, visual, and cross-modal recognition were all nearly with familiar objects; haptic and visual recognition were also excellent with unfamiliar objects, but cross-modal recognition was less accurate. In Experiment 3, cross-modal recognition was also less accurate than within-mode recognition with familiar objects that were members of the same basic-level category. The results indicate that children's haptic recognition is remarkably good, that cross-modal recognition is otherwise constrained, and that cross-modal recognition may be accomplished differently for familiar and unfamiliar objects.

Children's haptic and cross-modal recognition with familiar and unfamiliar objects.

Children's haptic and cross-modal recognition with familiar and unfamiliar objects.

Seeing through sound: dolphins (Tursiops truncatus) perceive the spatial structure of objects through echolocation.

Experiment 1 tested a dolphin (Tursiops truncatus) for cross-modal recognition of 25 unique pairings of 8 familiar, complexly shaped objects, using the senses of echolocation and vision. Cross-modal recognition was errorless or nearly so for 24 of the 25 pairings under both visual to echoic matching (V-E) and echoic to visual matching (E-V). First-trial recognition occurred for 20 pairings under V-E and for 24 under E-V. Echoic decision time under V-E averaged only 1.88 s. Experiment 2 tested 4 new pairs of objects for 24 trials of V-E and 24 trials of E-V without any prior exposure of these objects. Two pairs yielded performance significantly above chance in both V-E and E-V. Also, the dolphin matched correctly on 7 of 8 1st trials with these pairs. The results support a capacity for direct echoic perception of object shape by this species and demonstrate that prior object exposure is not required for spontaneous cross-modal recognition.

Categorical and Dimensional Decoding of Emotional Intonations in Patients with Focal Brain Lesions

The present study attempts to elucidate whether cerebral brain lesions differentially affect the crossmodal decoding of emotional intonations in semantically meaningless sentences. Forty patients with well-documented lesions and 12 matched hospital controls participated in the study. Twenty-one had left brain damage (LBD: 12 with anterorolandic (anterior) and 9 with retrorolandic-infrasylvian (posterior) lesions); 19 had right brain damage (RBD: 12 anterior, 7 posterior lesions). The decoding of emotioncategorieswas measured using (a) multiple choice of verbal labels and (b) matching one emotional vocalization (joy, fear, sadness, or anger) with two choice facial expressions. Crossmodaldimensionaldecoding was assessed by matching vocalizations with two facial expressions with regard to emotional valence or arousal. Results indicate that labeling was reduced in all lesion groups as compared to that in controls. Crossmodal categorical recognition was impaired in RBD, whereas LBD performance was comparable to controls. However, in the dimensional decoding task, a reduced recognition of valence in LBD and arousal in RBD was observed. An analysis of localizational subgroups revealed that subjects with left ventral frontal lesions, which in part extended into the adjacent right hemisphere, were predominantly impaired in the crossmodal identification of valence, whereas right temporoparietal lesions affected arousal decoding. Our results suggest that lateralized lesions may differentially affect the crossmodal recognition of dimensional concepts such as valence and arousal.

Effects of partial versus complete lesions of the amygdala on cross-modal associations in cynomolgus monkeys

Aspiration lesions of the amygdala plus subjacent cortex were found earlier to produce a severe impairment in cross-modal (tactual-to-visual) recognition (Murray & Mishkin, 1985). To determine whether more selective lesions would also produce this effect, we trained 8 naive cynomolgus monkeys on a tactual-visual version of delayed nonmatching-to-sample and then injected ibotenic acid bilaterally into either the basolateral (n = 4) or centromedial (n = 4) subdivisions of the amygdala. Neither of the excitotoxic lesions affected performance. In a second experiment, we aspirated the amygdala plus subjacent cortex in similarly trained monkeys. The performance of these animals fell significantly and remained substantially below preoperative levels despite extensive postoperative retraining. The findings suggest that the severe deficit in cross-modal recognition following the aspiration lesion is attributable to complete amygdala damage, damage to the subjacent cortex, or the two in combination.

The picture superiority effect in a cross-modality recognition task

Words and pictures were studied and recognition tests given in which each studied object was to be recognized in both word and picture format. The main dependent variable was the latency of the recognition decision. The purpose was to investigate the effects of study modality (word or picture), of congruence between study and test modalities, and of priming resulting from repeated testing. Experiments 1 and 2 used the same basic design, but the latter also varied retention interval. Experiment 3 added a manipulation of instructions to name studied objects, and Experiment 4 deviated from the others by presenting both picture and word referring to the same object together for study. The results showed that congruence between study and test modalities consistently facilitated recognition. Furthermore, items studied as pictures were more rapidly recognized than were items studied as words. With repeated testing, the second instance was affected by its predecessor, but the facilitating effect of picture-to-word priming exceeded that of word-to-picture priming. The finds suggest a two- stage recognition process, in which the first is based on perceptual familiarity and the second uses semantic links for a retrieval search. Common-code theories that grant privileged access to the semantic code for pictures or, alternatively, dual-code theories that assume mnemonic superiority for the image code are supported by the findings. Explanations of the picture superiority effect as resulting from dual encoding of pictures are not supported by the data.

Neural correlates of cross-modal recognition memory by 8-month-old human infants.

Neural correlates of cross-modal recognition memory by 8-month-old human infants.

Neural correlates of cross-modal recognition memory by 8-month-old human infants.

The neural correlates of cross-modal recognition memory were examined in 8-month-old infants by using event-related potentials. Testing began by having all Ss feel (but not see) an object for 60 s. Test trials then followed. Infants in Condition 1 received 15 presentations of a picture of the familiar object, followed by alternating pictures of that object and a novel object. Infants in Condition 2 received 15 presentations of a picture of the novel object, followed by the same test sequence as infants in Condition 1. Infants in Condition 3 were presented with 2 test trials during which looking times were recorded to pictures of the familiar and novel objects; they then received the same test sequence as infants in Conditions 1 and 2. Infants in Condition 4 were presented only with the same test sequence as infants in Conditions 1, 2, and 3. Only in Conditions 1 and 4 was a late positive slow wave invoked by the novel object (indicative of recognition memory), although infants in Condition 3 did show a significant looking preference for the novel object

Behavioral assessment of young nonhuman primates: Perceptual-cognitive development

The Infant Primate Research Laboratory at the University of Washington utilizes a range of test procedures to study perceptual-cognitive development in nonhuman primate infants at high and low risk for poor developmental outcome. These test procedures rely on the visual preferences of infants and are employed in research with both human and animal subjects. The forced-choice preferential looking technique is based on the primate infants' preference for patterned over plain stimuli and allows the early measurement of visual acuity. The novelty paradigm is based on the preference that infants show for novel over familiar stimuli and is used to study intramodal and cross-modal recognition memory. Tests of recognition memory have shown sensitivity in detecting subtle functional alterations associated with perinatal risk factors such as teratogen exposure. The use of measures which assess early patterns of visual attention should be considered in both human and nonhuman primate studies in behavioral teratology.

Spatial and verbal abilities in children's crossmodal recognition: A dual coding approach.

Spatial and verbal abilities in children's crossmodal recognition: A dual coding approach.

The neural structures involved in cross-modal recognition and tactile discrimination performance: An investigation using 2-DG

Given the failure of the ablation method to identify the neural structures/systems that are crucial for cross-modal recognition (CMR) and for tactile discrimination performance (TDP), we injected radioactive 2-deoxy-[14C]glucose (2-DG) into monkeys trained to a high level of CMR or TDP. Nine monkeys were trained to recognize in one sense-modality, to a level greater than 80% correct, the objects experienced in the alternate modality on only a single prior trial. After injection, CMR was continued (Expt. 1a) exactly as before, except that all CMR problems were now in only one direction; or (Expt. 1b) the second--the CMR--trial of each problem was not realized (i.e. no objects were available in the second modality) so that the monkey merely 'expected' the cross-modal trial. Nine other monkeys were trained on a graded roughness discrimination task with conventional 'titration' procedures to a stable level of performance, either (Expt. 2a) without having undergone any cortical removal, or (Expt. 2b) after unilateral removal--contralateral or ipsilateral to the preferred hand--of the posterior insula or of the second somatosensory projection cortex (SII). Exactly the same training procedures were continued in Expt. 2 after injection of 2-DG. Coronal radiographs were made at 490 microns for all 18 monkeys (and also for a 19th, which served as a control in Expt. 1b). The optical densities of the autoradiographs were measured quantitatively in respect of 50 structures or part-structures (e.g. sulci, thalamic nuclei, subcortical structures); they were rated either with the aid of 'pseudo-colours' produced by the computer, or directly by judging the black/white optical densities, in respect of 29 other structures. These 79 structures were then the dependent variables in MANOVAs or ANOVAs, to determine differences between groups; or within groups with respect to the left/right and ipsilateral/contralateral hemispheres; or interaction effects. It was found that certain structures (e.g. the ventral portion of the claustrum, the insula with its extension ventrally into the fundus, nucleus Medialis dorsalis and nucleus Pulvinar oralis) repeatedly, whereas others (e.g. the amygdala) never gave rise to significant outcomes. Structures in the left cerebral hemisphere were frequently found to be more strongly labelled than those in the right hemisphere.(ABSTRACT TRUNCATED AT 400 WORDS)

Cross-modal recognition of familiar and unfamiliar objects by the monkey: the effects of ablation of polysensory neocortex or of the amygdaloid complex.

Nine rhesus monkeys were trained to a standard level of cross-modal recognition (CMR) in the directions vision to touch and touch to vision. Their level of performance remained essentially unchanged with unfamiliar objects. Five then received bilateral removals of frontal, temporal and parietal polysensory cortex in one stage or successively, and 4 underwent removal of the amygdaloid complex in one stage. All animals were retrained to criterion with familiar objects and then again tested with unfamiliar objects. Postoperatively, the monkeys with extensive neocortical removals were unimpaired or slightly impaired with familiar objects, and slightly impaired (in only one direction) with unfamiliar objects. The animals with amygdaloid ablations showed a different pattern of change: with familiar objects they were unimpaired (if removals were less extensive), or were severely but transiently impaired (if the removals of the amygdala were more extensive and/or other structures were involved); with unfamiliar objects they were unimpaired, tending to improve. The neocortical polysensory areas may be necessary for generating new visual representations during learning--a performance required only for the CMR of unfamiliar objects.

Cross-modal recognition of pictures and descriptions without test-appropriate encoding

Droodles are simple line drawings that take on distinct and humorous meanings when coupled with appropriate descriptive phrases. Two experiments examined recognition memory for one element (drawing or phrase) when only the other had been studied earlier. In this way, the interpretive link between drawing and phrase could not be made at the time of encoding. Experiment 1 showed that people could recognize phrases when only the corresponding pictures had been studied, and Experiment 2 showed that people could recognize pictures when only the corresponding phrases had been studied. Thus, cross-modal recognition is successful when testappropriate encodings could not have been prepared during study. Also, performance was unrelated to imagery ability in both experiments. Apparently, pictures and phrases are encoded and stored so as to permit quite dramatic reinterpretation and comparison during retrieval.

Cross-modal recognition by the monkey: Can the experience in the first modality be reduced to a single trial?

Using the cross-modal recognition procedure previously described, but now progressively reducing the number of learning trials given in the first sense modality, three unselected and unoperated rhesus monkeys (and one with bilateral removals of frontal, temporal and parietal polysensory cortex) were able to maintain an average level of 75% correct cross-modal recognition with only 1–3 learning trials. The pool size seemed less important than the familiarity of the objects. No discontinuity between single-trial and multiple-trial learning procedures was detected. Instead, the number of objects available at one time during learning was regarded as critical for distinguishing between these two kinds of learning procedure.

Confusion of Laterally Inverted Mirror-Images: A Relationship to Brain Anatomy?

Whereas in the monkey brain the representation of spatial direction (left/right) is the same for visual and for tactual inflow, in the human brain visual and tactual lateral directions are not aligned with respect to one another. This ana tomical feature of the human brain may account in part for the particular difficulty young sighted children have with laterally inverted forms. A small group of children who were totally blind since birth distinguished mirror pairs by touch significantly more easily than did sighted children of comparable age; and monkeys succeeded at cross-modal recognition of laterally inverted mirror pairs better (relative to control pairs) than did sighted children.

Development of cross-modal recognition in infant pigtail monkeys (Macaca nemestrina).

Development of cross-modal recognition in infant pigtail monkeys (Macaca nemestrina).

Development of cross-modal recognition in infant pigtail monkeys (Macaca nemestrina).

Development of cross-modal recognition in infant pigtail monkeys (Macaca nemestrina).

The relationship between metaphorical and cross-modal abilities: Failure to demonstrate metaphorical recognition in chimpanzees capable of cross-modal recognition

Two chimpanzees, trained to recognise objects across the sense-modalities of vision and touch, failed to give evidence of metaphorical matching ability. Therefore in the chimpanzee cross-modal recognition is unlikely to be a sub-set of metaphorical matching.

Cross-modal recognition by the monkey: The effects of cortical removals

Two rhesus monkeys that were already highly proficient at cross-modal recognition (touch to vision and vision to touch) underwent successively bilateral removals of cortex from the superior temporal sulcus and from the dorso lateral prefrontal region. One animal acquired a persistent visual impairment after the first removal; and both were transiently severely impaired at tactile learning after the second removals. Nonetheless, both animals gave evidence of a high level of cross-modal recognition after each operation.