Cross-modal Abilities Research Articles

RF-CM

Radio-Frequency (RF) based human activity recognition (HAR) enables many attractive applications such as smart home, health monitoring, and virtual reality (VR). Among multiple RF sensors, mmWave radar is emerging as a new trend due to its fine-grained sensing capability. However, laborious data collection and labeling processes are required when employing a radar-based sensing system in a new environment. To this end, we propose RF-CM, a general cross-modal human activity recognition framework. The key enabler is to leverage the knowledge learned from a massive WiFi dataset to build a radar-based HAR system with limited radar samples. It can significantly reduce the overhead of training data collection. In addition, RF-CM can work well regardless of the deployment setups of WiFi and mmWave radar, such as performing environments, users' characteristics, and device deployment. RF-CM achieves this by first capturing the activity-related variation patterns through data processing schemes. It then employs a convolution neural network-based feature extraction module to extract the high-dimensional features to be fed into the activity recognition module. Finally, RF-CM takes the generalization knowledge from WiFi networks as guide labels to supervise the training of the radar model, thus enabling a few-shot radar-based HAR system. We evaluate RF-CM by applying it to two HAR applications, fine-grained American sign language recognition (WiFi-cross-radar) and coarse-grained gesture recognition (WiFi-cross-RFID). The accuracy improvement of over 10% in both applications demonstrates the effectiveness of RF-CM. This cross-modal ability allows RF-CM to support more cross-modal applications.

Cross-Talk of Low-Level Sensory and High-Level Cognitive Processing: Development, Mechanisms, and Relevance for Cross-Modal Abilities of the Brain.

The emergence of cross-modal learning capabilities requires the interaction of neural areas accounting for sensory and cognitive processing. Convergence of multiple sensory inputs is observed in low-level sensory cortices including primary somatosensory (S1), visual (V1), and auditory cortex (A1), as well as in high-level areas such as prefrontal cortex (PFC). Evidence shows that local neural activity and functional connectivity between sensory cortices participate in cross-modal processing. However, little is known about the functional interplay between neural areas underlying sensory and cognitive processing required for cross-modal learning capabilities across life. Here we review our current knowledge on the interdependence of low- and high-level cortices for the emergence of cross-modal processing in rodents. First, we summarize the mechanisms underlying the integration of multiple senses and how cross-modal processing in primary sensory cortices might be modified by top-down modulation of the PFC. Second, we examine the critical factors and developmental mechanisms that account for the interaction between neuronal networks involved in sensory and cognitive processing. Finally, we discuss the applicability and relevance of cross-modal processing for brain-inspired intelligent robotics. An in-depth understanding of the factors and mechanisms controlling cross-modal processing might inspire the refinement of robotic systems by better mimicking neural computations.

The effect of multimodal information on children’s numerical judgments

Although much research suggests that adults, infants, and nonhuman primates process number (among other properties) across distinct modalities, limited studies have explored children’s abilities to integrate multisensory information when making judgments about number. In the current study, 3- to 6-year-old children performed numerical matching or numerical discrimination tasks in which numerical information was presented either unimodally (visual only), cross-modally (comparing audio with visual), or bimodally (simultaneously presenting audio and visual input). In three experiments, we investigated children’s multimodal numerical processing across distinct task demands and difficulty levels. In contrast to previous work, results indicate that even the youngest children (3 and 4 years) performed above chance across all three modality presentations. In addition, the current study contributes two other novel findings, namely that (a) children exhibit a cross-modal disadvantage when numerical comparisons are easy and that (b) accuracy on bimodal trial types led to even more accurate numerical judgments under more difficult circumstances, particularly for the youngest participants and when precise numerical matching was required. Importantly, findings from this study extend the literature on children’s numerical cross-modal abilities to reveal that, like their adult counterparts, children readily track and compare visual and auditory numerical information, although their abilities to do so are not perfect and are affected by task demands and trial difficulty.

Hearables: Multimodal physiological in-ear sensing

Future health systems require the means to assess and track the neural and physiological function of a user over long periods of time, and in the community. Human body responses are manifested through multiple, interacting modalities – the mechanical, electrical and chemical; yet, current physiological monitors (e.g. actigraphy, heart rate) largely lack in cross-modal ability, are inconvenient and/or stigmatizing. We address these challenges through an inconspicuous earpiece, which benefits from the relatively stable position of the ear canal with respect to vital organs. Equipped with miniature multimodal sensors, it robustly measures the brain, cardiac and respiratory functions. Comprehensive experiments validate each modality within the proposed earpiece, while its potential in wearable health monitoring is illustrated through case studies spanning these three functions. We further demonstrate how combining data from multiple sensors within such an integrated wearable device improves both the accuracy of measurements and the ability to deal with artifacts in real-world scenarios.

Pathways From Toddler Information Processing to Adolescent Lexical Proficiency.

This study examined the relation of 3-year core information-processing abilities to lexical growth and development. The core abilities covered four domains-memory, representational competence (cross-modal transfer), processing speed, and attention. Lexical proficiency was assessed at 3 and 13 years with the Peabody Picture Vocabulary Test (PPVT) and verbal fluency. The sample (N = 128) consisted of 43 preterms (< 1750 g) and 85 full-terms. Structural equation modeling indicated concurrent relations of toddler information processing and language proficiency and, independent of stability in language, direct predictive links between (a) 3-year cross-modal ability and 13-year PPVT and (b) 3-year processing speed and both 13-year measures, PPVT and verbal fluency. Thus, toddler information processing was related to growth in lexical proficiency from 3 to 13 years.

Crossmodal Integration of Conspecific Vocalizations in Rhesus Macaques

Crossmodal integration of audio/visual information is vital for recognition, interpretation and appropriate reaction to social signals. Here we examined how rhesus macaques process bimodal species-specific vocalizations by eye tracking, using an unconstrained preferential looking paradigm. Six adult rhesus monkeys (3M, 3F) were presented two side-by-side videos of unknown male conspecifics emitting different vocalizations, accompanied by the audio signal corresponding to one of the videos. The percentage of time animals looked to each video was used to assess crossmodal integration ability and the percentages of time spent looking at each of the six a priori ROIs (eyes, mouth, and rest of each video) were used to characterize scanning patterns. Animals looked more to the congruent video, confirming reports that rhesus monkeys spontaneously integrate conspecific vocalizations. Scanning patterns showed that monkeys preferentially attended to the eyes and mouth of the stimuli, with subtle differences between males and females such that females showed a tendency to differentiate the eye and mouth regions more than males. These results were similar to studies in humans indicating that when asked to assess emotion-related aspects of visual speech, people preferentially attend to the eyes. Thus, the tendency for female monkeys to show a greater differentiation between the eye and mouth regions than males may indicate that female monkeys were slightly more sensitive to the socio-emotional content of complex signals than male monkeys. The current results emphasize the importance of considering both the sex of the observer and individual variability in passive viewing behavior in nonhuman primate research.

Profiling nonhuman intelligence: An exercise in developing unbiased tools for describing other “types” of intelligence on earth

Intelligence has historically been studied by comparing nonhuman cognitive and language abilities with human abilities. Primate-like species, which show human-like anatomy and share evolutionary lineage, have been the most studied. However, when comparing animals of non-primate origins our abilities to profile the potential for intelligence remains inadequate. Historically our measures for nonhuman intelligence have included a variety of tools: (1) physical measurements – brain to body ratio, brain structure/convolution/neural density, presence of artifacts and physical tools, (2) observational and sensory measurements – sensory signals, complexity of signals, cross-modal abilities, social complexity, (3) data mining – information theory, signal/noise, pattern recognition, (4) experimentation – memory, cognition, language comprehension/use, theory of mind, (5) direct interfaces – one way and two way interfaces with primates, dolphins, birds and (6) accidental interactions – human/animal symbiosis, cross-species enculturation. Because humans tend to focus on “human-like” attributes and measures and scientists are often unwilling to consider other “types” of intelligence that may not be human equated, our abilities to profile “types” of intelligence that differ on a variety of scales is weak. Just as biologists stretch their definitions of life to look at extremophiles in unusual conditions, so must we stretch our descriptions of types of minds and begin profiling, rather than equating, other life forms we may encounter.COMPLEX (COmplexity of Markers for Profiling Life in EXobiology) offers a new approach to profile a variety of organisms along multiple dimensions including EQ – Encephalization Quotient, CS – Communication Signal complexity, IC – Individual Complexity, SC – Social Complexity and II – Interspecies Interaction. Because Earth species are found along a variety of continuums, defining an intelligence profile along these different trajectories rather than comparing them only to human intelligence, may give us insight into a potential tool for quickly assessing unknown species. The application of profiling nonhuman species, out of world, will be both observational and potentially interactive in some way. Using profiles and indicators gleaned from Earth species to help us develop profiles and using pattern recognition, modeling and other data mining techniques could help jump start our understanding of other organisms and their potential for certain “types” of intelligence.

Interactions between Senses: Updating on Neural Mechanisms and Behavioral Evidence

Interactions between Senses: Updating on Neural Mechanisms and Behavioral Evidence

Musical Sight-Reading Expertise: Cross-Modality Investigations

Abstract. It is often said that experienced musicians are capable of hearing what they read from the score and vice versa. This suggests that they are able to process and to integrate multimodal information. Does the expertise in music rely on an efficient cross-modal integration? This paper investigates this issue with 2 experiments. In the first one, 30 expert and 31 non expert musicians were required to report whether two successively auditory and visual presented fragments of classical music were same or different. In half the conditions the participants received the fragments in visual presentation only (same modal presentation), in the other half they received the fragments in auditory and visual presentation (cross-modal presentation). As expected, analysis of Response Time and Errors showed that experts performed the task more accurately and rapidly than non experts, whatever the modal presentation, while non experts performed more accurately and rapidly in the same modal presentation. So, more experienced performers seem to be better able to transfer information from one modality to another. In the second experiment, 15 expert and 10 non expert musicians were required to listen, read, and perform classical piano excerpts. The experiment was run in two consecutive phases during which each excerpt was (1) read without playing and (2) sight-read (read and played). In half the conditions, the participants heard the music before the reading phases. The excerpts contained suggested fingering of variable difficulty (difficult, easy, or no fingering). Eye movements were registered during the reading phases. Analyses of fixations and playing mistakes validated the hypothesized modal independence of information among expert musicians, observed in the first experiment. Moreover analyses validated the cross-modal capacities of expert memory. For experts, the mere viewing of a musical score may facilitate planning and preparation for motor execution, whereas non-experts do not appear to have this cross-modal integration ability. Results are discussed in terms of amodal memory for expert musicians that can be in support of theoretical work by Ericsson and Kintsch [24]: more experienced performers better integrate knowledge across modalities

Tactual, Visual, and Cross-Modal Transfer of Texture in 5- and 8-Year-Old Children

Children's tactual, visual, and cross-modal transfer abilities for texture were investigated in a delayed matching-to-sample paradigm. Transfer performance from vision to touch was found to increase between 5 and 8 years of age, whereas transfer performance from touch to vision did not vary with age and matched touch-to-touch performance. Asymmetrical cross-modal abilities were observed at the age of 8 years, vision-to-touch transfer performance being higher than touch-to-vision transfer performance (experiment 2). This developmental pattern could not be attributed to limitations in the tactual or visual discriminability of the textures or to differences in tactual or visual memory between the two age groups (experiment 1). It is suggested that the increase with age in vision-to-touch performance may be related to the intervention of more efficient top-down perceptual processes in the older children.

Visual and Auditory Temporal Processing, Cross-Modal Transfer, and Reading

The present study reexamined the relevance of auditory and visual cross-modal matching to reading ability, an issue first addressed in a seminal study by Birch and Belmont (1964). By presenting all patterns to be matched as temporal sequences of tones and lights, including intramodal as well as cross-modal conditions, and covarying memory, three problems with the Birch and Belmont design were corrected. Results showed that poor readers had difficulty in perceiving temporal patterns generally: They did worse than good readers not only on cross-modal conditions but also on intramodal ones. These results were replicated in two tasks. Nonetheless, hierarchical regressions provided some indication that cross-modal abilities themselves are relevant to reading. For one of the two tasks, cross-modal performance contributed to the prediction of reading ability over and above intramodal performance. Poor readers also showed slower response times--a factor that contributed marginally to the prediction of reading independent of temporal processing.

Continuity in tactual-visual cross-modal transfer: infancy to 11 years.

The present study demonstrated that individual differences in cross-modal transfer showed continuity over a 10-year span. Tactual-visual tasks, requiring visual recognition of shapes that had previously been felt but not seen, were given to full-term and preterm children at 2 ages, 1 and 11 years. Cross-modal performance showed a left-hand advantage at 11 years and, for both groups, cross-age correlations were significant when tactual exploration at 11 years was done with the left hand (r = .34-.36). The continuity showed some specificity in that the infant measure did not relate to other types of cross-modal performance at 11 years and was not dependent on aspects of spatial ability involving form perception. This continuity accounted for most of the previously reported relation of infant cross-modal ability to 11-year IQ.

Apes and language: Human uniqueness again?

AbstractWilkins &amp; Wakefield's intriguing model of language evolution is deficient in evidence of human uniqueness in metaphorical matching, amodal representation, reference, conceptual structure, hierarchical organization, linguistic comprehension, sign use, laterality, and handedness. Primates show communicative reference, laterality, and handedness, and apes in particular show hierarchical organization, conceptual structure, cross-modal abilities, sign use, and displaced reference.

Brief report: Preferential looking in intermodal perception by children with autism

Recent reviews of autism have debated whether autism should beconsidered primarily as a disorder of affective and social relations asKanner proposed (see Hobson, 1989) or as a cognitive deficit, specificallyin the ability to construct a "theory of mind" (Baron-Cohen, Leslie, F Leslie, 1987). As Ozonoff, Pennington, and Rogers (1990)have noted, to answer basic questions about autism we must determinewhich features of autism are universal and primary. Consequently, we mustdevelop tasks that can be used irrespective of age or verbal abilities. Thestudy reported here is a first step towards extending methods used withnormal infants to examine the functioning of children with autism. Suchprocedures should be particularly well suited for use with children withautism because the tasks do not require sophisticated responses and aretypically less intrusive.An intermodal preference task was selected for the present study be-cause of its utility in the study of infants' cross-modal abilities and emotionperception, both of which have been implicated in the syndrome of autism(Bryson, 1972; Hobson, Ouston, & Lee, 1988). The intermodal preferencemethod combines the potential for presenting dynamic, multimodal events

The relationship between metaphoric and crossmodal abilities during development.

Metaphoric and crossmodal abilities were investigated in children, aged 3-8 years. For this study metaphoric performance was regarded as based on the production of a relationship between physically dissimilar stimuli (presented in two sense-modalities), which are not normally perceived as being related. It was asked at what age crossmodal performance, based on specific rather than global/categorical differences within stimulus pairs, can be regarded as distinct from metaphoric performance. In Expt. 1, metaphoric and crossmodal two-trial "recognition" tasks were given in Touch and Vision at all ages. In Expt. 2,3- and 4-year-olds were first trained over 6-20 trials to choose one of two stimuli in one modality (Touch or Vision) and then given one "recognition" trial in the alternate modality. Children as young as three could relate specific physical stimulus differences across sense-modalities. Although metaphoric recognition was present on two problems in children aged 3 and 4 years, metaphoric performance was inferior to crossmodal performance at all ages. Crossmodal abilities improved with age due to improved intramodal skills; such intramodal improvement can probably account for only a part of the improvement on the metaphoric task. Reasons are given why metaphoric and crossmodal abilities may be separate over the age range 3-8 years.

The relationship between metaphorical and cross-modal abilities: Failure to demonstrate metaphorical recognition in chimpanzees capable of cross-modal recognition

Two chimpanzees, trained to recognise objects across the sense-modalities of vision and touch, failed to give evidence of metaphorical matching ability. Therefore in the chimpanzee cross-modal recognition is unlikely to be a sub-set of metaphorical matching.

Intra- and Cross-Modal Transfer in Relation to Brain Dysfunction and Language Disorder

Brain damaged and normal subjects were tested on their ability to transfer information of varying complexity across sensory modalities. Subjects without neurological damage were compared to subjects with left and right hemisphere lesions. Tasks varied in complexity from simple matching of Morse code patterns within a sensory modality to cross-modal transfer across sensory modalities. The results confirmed that cross-modal matching ability is significantly impaired in subjects with posterior left hemisphere lesions, whereas subjects with lesions in other areas of the brain show minimal deficits in cross-modal abilities. The results are interpreted to support a multimodal basis for complex intellectual function. Language is hypothesized to be one response capability that depends on intact cross-modal functions, with the angular gyrus region of the dominant hemisphere forming an anatomical basis.

Information Matching within and between Auditory and Visual Sense Modalities and Mathematics Achievement

The ability to integrate information arriving through different sense modalities has been examined closely in regard to reading performance, but research in mathematics learning has largely ignored this ability. In the present study, 180 fourth-grade children stratified according to IQ were administered four tests designed to measure both intramodal and cross-modal auditory and visual matching ability. Performances on these measures were correlated with achievement measures in reading and mathematics. Correlations significant at the 0.01 and 0.05 levels (df=58) were found for both reading and mathematics, but those for mathematics were consistently higher. The variation of the correlations with mathematics achievement over the IQ levels was rather striking, indicating the potential use of sensory matching ability in research on individualization or in ATI experiments.

Recognition of shapes across modalities by infants.

HUMAN adults and children as young as four years can recognize two identical objects as equivalent when they touch one and see the other1–3. It is not yet known what mechanisms underlie this cross-modal ability. One suggestion4–6 has been that humans are able to match identical objects cross-modally because they can give the visual and tactual objects the same name. This hypothesis has recently been questioned7, and an experiment which showed that chimpanzees and orang-outangs can to some extent match shapes cross-modally8 makes it improbable that cross-modal mechanisms need always be based on language. But it may still be true that humans have to be able to attach common verbal labels to visual and tactual inputs in order to treat them as the same.

Comments on the Design of Developmental Studies of Cross-Modal Matching and Cross-Modal Transfer

It has been suggested in several recent studies that there is an increase with age in the ability to treat the same stimulus as equivalent when information about this stimulus is presented through two different modalities. However, the improvement in cross-modal performance in these studies can be explained more simply in terms of a corresponding improvement in the ability to discriminate within modalities. This alternative explanation has not been satisfactorily excluded. Suggestions are made for the correct design of developmental studies of cross-modal abilities.