Multisensory Recognition Research Articles

ND-MRM: Neuronal Diversity Inspired Multisensory Recognition Model

Cross-sensory interaction is a key aspect for multisensory recognition. Without cross-sensory interaction, artificial neural networks show inferior performance in multisensory recognition. On the contrary, the human brain has an inherently remarkable ability in multisensory recognition, which stems from the diverse neurons that exhibit distinct responses to sensory inputs, especially the multisensory neurons with multisensory responses hence enabling cross-sensory interaction. Based on this neuronal diversity, we propose a Neuronal Diversity inspired Multisensory Recognition Model (ND-MRM), which, similar to the brain, comprises unisensory neurons and multisensory neurons. To reflect the different responses characteristics of diverse neurons in the brain, special connection constraints are innovatively designed to regulate the features transmission in the ND-MRM. Leveraging this novel concept of neuronal diversity, our model is biologically plausible, enabling more effective recognition of multisensory information. To validate the performance of the proposed ND-MRM, we employ a multisensory emotion recognition task as a case study. The results demonstrate that our model surpasses state-of-the-art brain-inspired baselines on two datasets, proving the potential of brain-inspired methods for advancing multisensory interaction and recognition.

The Onset of Maternal Behavior in Sheep and Goats: Endocrine, Sensory, Neural, and Experiential Mechanisms.

In sheep and goats, the onset of maternal behavior at parturition is characterized by a first phase called maternal responsiveness during which the mother is attracted to any newborn. In a second phase, called maternal selectivity, the mother establishes a selective bond with her young so that she only accepts it at suckling. After a description of the behavioral expression of both phases, this chapter reviews the physiological, sensory, and neural mechanisms involved. These two behavioral processes are synchronized with parturition by the vaginocervical stimulation induced by the expulsion of the newborn. Olfactory cues provided by the neonate are involved in maternal responsiveness and selectivity. Oxytocin supported by estrogens is the key factor for maternal responsiveness. The neural network involved in maternal responsiveness is mainly hypothalamic and is different from the circuitry involved in selectivity, which mainly concerns olfactory processing regions. Visual and auditory cues are necessary for offspring recognition at a distance. This multisensory recognition suggests that mothers form a mental image of their young. Maternal experience renders mothers more responsive to maternally relevant physiology and to young-related sensory inputs.

Energy scavenging artificial nervous system for detecting rotational movement

The human sensory nervous system recognizes the change in the external environment and transmits the information to the brain. The brain uses information from the vestibular system in the head, which plays a vital role in the multisensory recognition of balance and orientation. The mimicking of human neural network systems based on biological synapses has emerged as a key technology in bioinspired electronics. This study emulates an energy scavenging artificial nervous system for detecting rotational movement to mimic the functions and mechanisms of the biological semicircular canals. Synaptic transistors realize the biological synaptic plasticity, which can be used to simulate short-to long-term memory transitions and learning processes. Furthermore, by integrating synaptic transistors with triboelectric rotation sensors, we have demonstrated that our artificial nervous system is able to detect rotational movements on the coordinates in real time. This artificial nervous system has promising applications including neurorobotics, soft electronics, and neuroprosthetics.

FMCW radar and inertial sensing synergy for assisted living

This study presents preliminary results about the multi-sensory recognition of indoor daily activities and fall detection, to monitor the well-being of older people at risk of physical and cognitive chronic health conditions. Five different sensors, continuous wave (CW) radar, frequency-modulated CW (FMCW) radar, and inertial measurement unit comprising an accelerometer, gyroscope, and magnetometer were used to simultaneously collect data from 20 subjects performing 10 activities. Rather than using all of the available sensors, it is more efficient and economical to select part of them to maximise the classification accuracy and avoid unnecessary computation to process information if it is not salient. Each individual sensor and several sensor combinations are trained with a quadratic-kernel support vector machine classifier. In addition, they are validated with an improved statistical approach, which uses data from unknown participants to test model rather than random cross-validation to verify if the model generalises well for unknown subjects. Furthermore, the most suitable sensor combinations are derived for each specific group of tested subjects selected (e.g. the oldest, youngest, tallest, and shortest sub-groups of participants out of the entire group).

Individual recognition through olfactory-auditory matching in lemurs.

Individual recognition can be facilitated by creating representations of familiar individuals, whereby information from signals in multiple sensory modalities become linked. Many vertebrate species use auditory-visual matching to recognize familiar conspecifics and heterospecifics, but we currently do not know whether representations of familiar individuals incorporate information from other modalities. Ring-tailed lemurs (Lemur catta) are highly visual, but also communicate via scents and vocalizations. To investigate the role of olfactory signals in multisensory recognition, we tested whether lemurs can recognize familiar individuals through matching scents and vocalizations. We presented lemurs with female scents that were paired with the contact call either of the female whose scent was presented or of another familiar female from the same social group. When the scent and the vocalization came from the same individual versus from different individuals, females showed greater interest in the scents, and males showed greater interest in both the scents and the vocalizations, suggesting that lemurs can recognize familiar females via olfactory-auditory matching. Because identity signals in lemur scents and vocalizations are produced by different effectors and often encountered at different times (uncoupled in space and time), this matching suggests lemurs form multisensory representations through a newly recognized sensory integration underlying individual recognition.

Multisensory recognition of actively explored objects.

Shape recognition can be achieved through vision or touch, raising the issue of how this information is shared across modalities. Here we provide a short review of previous findings on cross-modal object recognition and we provide new empirical data on multisensory recognition of actively explored objects. It was previously shown that, similar to vision, haptic recognition of objects fixed in space is orientation specific and that cross-modal object recognition performance was relatively efficient when these views of the objects were matched across the sensory modalities (Newell, Ernst, Tjan, & Bülthoff, 2001). For actively explored (i.e., spatially unconstrained) objects, we now found a cost in cross-modal relative to within-modal recognition performance. At first, this may seem to be in contrast to findings by Newell et al. (2001). However, a detailed video analysis of the visual and haptic exploration behaviour during learning and recognition revealed that one view of the objects was predominantly explored relative to all others. Thus, active visual and haptic exploration is not balanced across object views. The cost in recognition performance across modalities for actively explored objects could be attributed to the fact that the predominantly learned object view was not appropriately matched between learning and recognition test in the cross-modal conditions. Thus, it seems that participants naturally adopt an exploration strategy during visual and haptic object learning that involves constraining the orientation of the objects. Although this strategy ensures good within-modal performance, it is not optimal for achieving the best recognition performance across modalities.