Cross-modal Perception Research Articles

CMPNet: A cross-modal multi-scale perception network for RGB-T crowd counting

The cross-modal crowd counting method demonstrates better scene adaptability under complex conditions by introducing independent supplementary information. However, existing methods still face problems such as insufficient fusion of modal features, underutilization of crowd structure, and the neglect of scale information. In response to the above issues, this paper proposes a cross-modal multi-scale perception network (CMPNet). Specifically, CMPNet mainly consists of a cross-modal perception fusion module and a multi-scale feature aggregation module. The cross-modal perception fusion module effectively suppresses noise features while sharing features between different modalities, thereby significantly improving the robustness of the crowd counting process. The multi-scale feature aggregation module obtains rich crowd structure information through a spatial context aware graph convolution unit, and then integrates feature information from different scales to enhance the network’s perception ability of crowd density. To the best of our knowledge, CMPNet is the first attempt to model the crowd structure and mine its semantics in the field of cross-modal crowd counting. The experimental results show that CMPNet achieves state-of-the-art performance on all RGB-T datasets, providing an effective solution for cross-modal crowd counting. We will release the code at https://github.com/KunChenKKK/CMPNet.

Precision-based causal inference modulates audiovisual temporal recalibration.

Cross-modal temporal recalibration guarantees stable temporal perception across everchanging environments. Yet, the mechanisms of cross-modal temporal recalibration remain unknown. Here, we conducted an experiment to measure how participants' temporal perception was affected by exposure to audiovisual stimuli with consistent temporal delays. Consistent with previous findings, recalibration effects plateaued with increasing audiovisual asynchrony and varied by which modality led during the exposure phase. We compared six observer models that differed in how they update the audiovisual temporal bias during the exposure phase and whether they assume modality-specific or modality-independent precision of arrival latency. The causal-inference observer shifts the audiovisual temporal bias to compensate for perceived asynchrony, which is inferred by considering two causal scenarios: when the audiovisual stimuli have a common cause or separate causes. The asynchrony-contingent observer updates the bias to achieve simultaneity of auditory and visual measurements, modulating the update rate by the likelihood of the audiovisual stimuli originating from a simultaneous event. In the asynchrony-correction model, the observer first assesses whether the sensory measurement is asynchronous; if so, she adjusts the bias proportionally to the magnitude of the measured asynchrony. Each model was paired with either modality-specific or modality-independent precision of arrival latency. A Bayesian model comparison revealed that both the causal-inference process and modality-specific precision in arrival latency are required to capture the nonlinearity and asymmetry observed in audiovisual temporal recalibration. Our findings support the hypothesis that audiovisual temporal recalibration relies on the same causal-inference processes that govern cross-modal perception.

Cross-modal representation of chewing food in posterior parietal and visual cortex.

Even though the oral cavity is not visible, food chewing can be performed without damaging the tongue, oral mucosa, or other intraoral parts, with cross-modal perception of chewing possibly critical for appropriate recognition of its performance. This study was conducted to clarify the relationship of chewing food cross-modal perception with cortex activities based on examinations of the posterior parietal cortex (PPC) and visual cortex during chewing in comparison with sham chewing without food, imaginary chewing, and rest using functional near-infrared spectroscopy. Additionally, the effects of a deafferent tongue dorsum on PPC/visual cortex activities during chewing performance were examined. The results showed that chewing food increased activity in the PPC/visual cortex as compared with imaginary chewing, sham chewing without food, and rest. Nevertheless, those activities were not significantly different during imaginary chewing or sham chewing without food as compared with rest. Moreover, subjects with a deafferent tongue dorsum showed reduced PPC/visual cortex activities during chewing food performance. These findings suggest that chewing of food involves cross-modal recognition, while an oral somatosensory deficit may modulate such cross-modal activities.

Cross-modal perception of puppies and adult conspecifics in dogs (Canis familiaris).

Cross-modal perception of puppies and adult conspecifics in dogs (Canis familiaris).

Supplemental Material for Cross-modal Perception of Puppies and Adult Conspecifics in Dogs (Canis familiaris)

Supplemental Material for Cross-modal Perception of Puppies and Adult Conspecifics in Dogs (Canis familiaris)

Analysis of the sweetening effects of lactone and ketone compounds in a cheddar cheese matrix using sensory analysis and molecular simulation

This study evaluated the effects of lactone and ketone compounds on sweetness perception in a Cheddar cheese matrix. According to the static sensory evaluation, a ternary mixture of γ-decanolactone/δ-decanolactone/acetoin at middle and high concentrations exhibited a significant sweetening effect closely related to olfactory input. Dynamic sensory evaluation showed that a γ-decanolactone/δ-decanolactone/acetoin ternary mixture at middle and high concentrations could enhance the sweetness perception of a Cheddar cheese matrix at different sucrose concentrations during oral processing. Molecular docking showed that the ternary mixture of γ-decanolactone/δ-decanolactone/acetoin with sucrose-T1R2 increased the number of hydrogen bond binding sites and increased the number of hydrophobic interaction sites. Molecular dynamics suggested that the molecular conformation of sucrose and T1R2 was more compact and stable in the ternary mixture. Additionally, in the presence of the ternary system, the binding free energy of the re-docking of sucrose and T1R2 was −7.28 kcal/mol, indicating that the docking structure is stable. This stability is maintained by the hydrogen bonding and hydrophobic interaction between the T1R2-sucrose complex and aroma mixtures. These findings provide a theoretical basis for using cross-modal perception to sweeten Cheddar cheese to cater to consumers’ pursuit of sweetness.

Oral processing of bakery products: An overview of current status and future outlook

Food oral processing (FOP) is an emerging research topic that allows a better comprehension of the relation between intrinsic food factors (physicochemical and sensory properties), and human physiology and eating behaviours. FOP can then help in the design of novel and healthier food to meet both quality requirements and consumer needs. In this context, this review presents the current state of knowledge and new insights for future research concerning FOP of bakery products. The application of FOP in bakery science is new, with different protocols and related evaluation being applied, as detailed in this review. The current knowledge shows that bread structure and texture, influenced by formulation and process conditions, as well as the crust and shortenings present, impact bread breakdown and bolus formation, sensory perception, and food physiological effects such as glycaemic response, satiation, and satiety. For a better comprehension of oral processing, cross-modal perception between sensory attributes is used to understand consumer perception and this is carried out using both in vivo, and in vitro methods. This review highlighted the great potential of FOP to assist researchers and producers to face the current challenges relating to i – salt, fat and sugar reduction in bakery products to fulfil current food nutrition policies, ii – the design of healthier bakery products, and iii – the development of bakery products for consumers with special dietary requirements and ageing needs. In conclusion, FOP shows great potential to assist in the development of novel and healthier foods to meet actual food nutrition policies and consumer needs and should be more explored in bakery science and production.

Robust Audio-Visual Contrastive Learning for Proposal-Based Self-Supervised Sound Source Localization in Videos.

By observing a scene and listening to corresponding audio cues, humans can easily recognize where the sound is. To achieve such cross-modal perception on machines, existing methods take advantage of the maps obtained by interpolation operations to localize the sound source. As semantic object-level localization is more attractive for prospective practical applications, we argue that these map-based methods only offer a coarse-grained and indirect description of the sound source. Additionally, these methods utilize a single audio-visual tuple at a time during self-supervised learning, causing the model to lose the crucial chance to reason about the data distribution of large-scale audio-visual samples. Although the introduction of Audio-Visual Contrastive Learning (AVCL) can effectively alleviate this issue, the contrastive set constructed by randomly sampling is based on the assumption that the audio and visual segments from all other videos are not semantically related. Since the resulting contrastive set contains a large number of faulty negatives, we believe that this assumption is rough. In this paper, we advocate a novel proposal-based solution that directly localizes the semantic object-level sound source, without any manual annotations. The Global Response Map (GRM) is incorporated as an unsupervised spatial constraint to filter those instances corresponding to a large number of sound-unrelated regions. As a result, our proposal-based Sound Source Localization (SSL) can be cast into a simpler Multiple Instance Learning (MIL) problem. To overcome the limitation of random sampling in AVCL, we propose a novel Active Contrastive Set Mining (ACSM) to mine the contrastive sets with informative and diverse negatives for robust AVCL. Our approaches achieve state-of-the-art (SOTA) performance when compared to several baselines on multiple SSL datasets with diverse scenarios.

Visual Mental Imagery and Neural Dynamics of Sensory Substitution in the Blindfolded Subjects

Although one can recognize the environment by soundscape substituting vision to auditory signal, whether subjects could perceive the soundscape as visual or visual-like sensation has been questioned. In this study, we investigated hierarchical process to elucidate the recruitment mechanism of visual areas by soundscape stimuli in blindfolded subjects. Twenty-two healthy subjects were repeatedly trained to recognize soundscape stimuli converted by visual shape information of letters. An effective connectivity method called dynamic causal modeling (DCM) was employed to reveal how the brain was hierarchically organized to recognize soundscape stimuli. The visual mental imagery model generated cortical source signals of five regions of interest better than auditory bottom-up, cross-modal perception, and mixed models. Spectral couplings between brain areas in the visual mental imagery model were analyzed. While within-frequency coupling is apparent in bottom-up processing where sensory information is transmitted, cross-frequency coupling is prominent in top-down processing, corresponding to the expectation and interpretation of information. Sensory substitution in the brain of blindfolded subjects derived visual mental imagery by combining bottom-up and top-down processing.

Perception of charisma in text and speech

Abstract The perception of leaders as charismatic personalities has been linked to the level of (positive) emotion in their messages. The present paper reports a cross-modal perception study on the relationship between perceived charisma and positive as well as negative emotions. One hundred forty-nine participants listened or read Brexit speeches by four British politicians (David Cameron, Nicola Sturgeon, Nigel Farage, Theresa May) and rated their charisma using a 7-point Likert scale. Emotions in speeches were quantified on three dimensions (valence, arousal, dominance) and supplemented by analyses of person deixis (I vs. we). Results revealed that effects of emotions on perceived charisma are moderated by the modality of speeches. Emotionally positive words as well as inclusive person deixis increased charisma ratings in written messages, but the effect was reduced or not present in auditory versions of these messages. Implications arise for studies of political discourse that tend to focus on scripted speeches.

Optoelectronic Synapses Based on MXene/Violet Phosphorus van der Waals Heterojunctions for Visual-Olfactory Crossmodal Perception

HighlightsThe photoelectric response of violet phosphorus (VP) is significantly enhanced by the MXene/VP van der Waals heterojunctions and the highest photoresponsivity of VP is demonstrated.The first VP-based optoelectronic synapse with essential synaptic behaviours is demonstrated.Proof-of-concept visual-olfactory crossmodal perception based on MXene/VP optoelectronic synapses is explored to mimic neuromorphic vision with multi-sensory interactions.

May the force be with you: exploring force discrimination in chimpanzees using the force-feedback device.

While force-feedback devices have been developed in areas such as virtual reality, there have been very few comparative cognitive studies in nonhuman animals using these devices. In addition, although cross-modal perception between vision and touch has been actively studied in nonhuman primates for several decades, there have been no studies of their active haptic perception. In this study, we attempted to train force discrimination in chimpanzees using a force-feedback device modified from a trackball. Chimpanzees were given different levels of force feedback (8.0 vs. 0.5N) when moving the on-screen cursor to the target area by manipulating the trackball and were required to select one of two choice stimuli based on the force cue. The experiment was conducted using a trial-block procedure in which the same force stimulus was presented for a fixed number of trials, and the force stimulus was changed between blocks. The block size was progressively reduced from ten trials. Four chimpanzees were trained, but none reached the learning criterion (80% or more correct responses under the condition that the force stimuli were presented randomly). However, a detailed analysis of the chimpanzees' performance before and after the trial-block switching revealed that their choice behavior could not be explained by a simple win-stay/lose-shift strategy, suggesting that the switching of the force stimuli affected the chimpanzees' choice behavior. It was also found that the chimpanzees performed better when switching from small to large force stimuli than when switching from large to small force stimuli. Although none of the chimpanzees in this study acquired force discrimination, future studies using such force-feedback devices will provide new insights for understanding haptic cognition in nonhuman primates from a comparative cognitive perspective.

Hearing temperatures: employing machine learning for elucidating the cross-modal perception of thermal properties through audition.

People can use their sense of hearing for discerning thermal properties, though they are for the most part unaware that they can do so. While people unequivocally claim that they cannot perceive the temperature of pouring water through the auditory properties of hearing it being poured, our research further strengthens the understanding that they can. This multimodal ability is implicitly acquired in humans, likely through perceptual learning over the lifetime of exposure to the differences in the physical attributes of pouring water. In this study, we explore people's perception of this intriguing cross modal correspondence, and investigate the psychophysical foundations of this complex ecological mapping by employing machine learning. Our results show that not only can the auditory properties of pouring water be classified by humans in practice, the physical characteristics underlying this phenomenon can also be classified by a pre-trained deep neural network.

Unsupervised Modality-Transferable Video Highlight Detection With Representation Activation Sequence Learning.

Identifying highlight moments of raw video materials is crucial for improving the efficiency of editing videos that are pervasive on internet platforms. However, the extensive work of manually labeling footage has created obstacles to applying supervised methods to videos of unseen categories. The absence of an audio modality that contains valuable cues for highlight detection in many videos also makes it difficult to use multimodal strategies. In this paper, we propose a novel model with cross-modal perception for unsupervised highlight detection. The proposed model learns representations with visual-audio level semantics from image-audio pair data via a self-reconstruction task. To achieve unsupervised highlight detection, we investigate the latent representations of the network and propose the representation activation sequence learning (RASL) module with k-point contrastive learning to learn significant representation activations. To connect the visual modality with the audio modality, we use the symmetric contrastive learning (SCL) module to learn the paired visual and audio representations. Furthermore, an auxiliary task of masked feature vector sequence (FVS) reconstruction is simultaneously conducted during pretraining for representation enhancement. During inference, the cross-modal pretrained model can generate representations with paired visual-audio semantics given only the visual modality. The RASL module is used to output the highlight scores. The experimental results show that the proposed framework achieves superior performance compared to other state-of-the-art approaches.

A Different Dimension in Evaluating the Relationship Between Space and Taste; Crossmodal Perception

Bu çalışma, mekân ve tat duyusunu crossmodal algı çerçevesinde incelemeyi amaçlamaktadır. Crossmodal algı, bir duyusal özelliğin başka bir duyusal özellikle ilişkilendirilme eğilimini gösteren bir olgu olan algıyı ifade etmektedir. Crossmodal algı, tatla ilişkili olarak tat-ses, tat-koku, tat-ışık vb. birçok duyusal benzeşme ile gerçekleşmektedir. Çalışma, crossmodal algının mekân tasarımı disiplini alanlarındaki yerini genişletmeye yöneliktir. Çalışmada, meta analiz yöntemi kullanılarak çeşitli literatür kaynaklarından elde edilen veriler analiz edilmiştir. Meta analizi sonucunda elde edilen bulgular göstermektedir ki; crossmodal algı, ağırlıklı olarak psikoloji alanında çalışılmakta fakat etkileri, mekân tasarım çerçevesini de etkilemektedir. Crossmodal algı çalışmalarının mekân tasarımı üzerindeki etkilerinin incelenmesi, kullanıcı deneyiminin yalnızca görsel ve işitsel boyutlarla sınırlı olmadığını; aynı zamanda çok katmanlı ve bütünsel bir duyusal deneyim olarak ele alınması gerektiğini ortaya koymaktadır. Çalışmanın sonucunda, mekân ve tat ilişkisinin değerlendirilmesine farklı bir boyut getirilmiş; crossmodal algı çalışmalarının etkilerinin mekân tasarımını da etkilediği ve mekân tasarımının bu algı çerçevesinde de ele alınması gerektiği vurgulanmıştır.

Cross-modal self-attention mechanism for controlling robot volleyball motion.

The emergence of cross-modal perception and deep learning technologies has had a profound impact on modern robotics. This study focuses on the application of these technologies in the field of robot control, specifically in the context of volleyball tasks. The primary objective is to achieve precise control of robots in volleyball tasks by effectively integrating information from different sensors using a cross-modal self-attention mechanism. Our approach involves the utilization of a cross-modal self-attention mechanism to integrate information from various sensors, providing robots with a more comprehensive scene perception in volleyball scenarios. To enhance the diversity and practicality of robot training, we employ Generative Adversarial Networks (GANs) to synthesize realistic volleyball scenarios. Furthermore, we leverage transfer learning to incorporate knowledge from other sports datasets, enriching the process of skill acquisition for robots. To validate the feasibility of our approach, we conducted experiments where we simulated robot volleyball scenarios using multiple volleyball-related datasets. We measured various quantitative metrics, including accuracy, recall, precision, and F1 score. The experimental results indicate a significant enhancement in the performance of our approach in robot volleyball tasks. The outcomes of this study offer valuable insights into the application of multi-modal perception and deep learning in the field of sports robotics. By effectively integrating information from different sensors and incorporating synthetic data through GANs and transfer learning, our approach demonstrates improved robot performance in volleyball tasks. These findings not only advance the field of robotics but also open up new possibilities for human-robot collaboration in sports and athletic performance improvement. This research paves the way for further exploration of advanced technologies in sports robotics, benefiting both the scientific community and athletes seeking performance enhancement through robotic assistance.

Scientific Understanding of the Desire to Touch and Haptic Presentation Methods Based on Cross-modal Perception--Toward New Horizons in Haptic Content That People Really Want to Touch

Scientific Understanding of the Desire to Touch and Haptic Presentation Methods Based on Cross-modal Perception--Toward New Horizons in Haptic Content That People Really Want to Touch

The Impact of Public Green Space Views on Indoor Thermal Perception and Environment Control Behavior of Residents - A Survey Study in Shanghai

To reduce energy consumption while meeting the indoor thermal comfort requirements of residential buildings, this survey study explores the visual-thermal cross-modal effect of public green space window view on residents’ indoor thermal perception, and how this effect regulates residents’ environment control behavior. Based on an online questionnaire, 424 valid data were collected from Shanghai residents during the lockdown period during which the window view is the only accessible connection to the outside. The result shows that public green space views, while not significantly changing the physical thermal sensation of residents, made a greater proportion of participants feel thermal neutral, and significantly improved the participants' environmental evaluation, thermal comfort, and thermal acceptance, resulting in a 7% decrease in using air conditioning and 7% increase in natural ventilation. Path analysis reveals that public green space window view can improve indoor thermal comfort by enhancing subjective environmental evaluation, and thus encourage passive environmental regulation behavior, reducing energy consumption. Complementing previous studies, this research provides direct evidence for the energy-saving potential of public green space in residential areas from a different angle.
 Keywords: Visual-thermal interaction; cross-modal perception; Environment control behavior; Environmental psychology;
 Sustainable design

Physicochemical features partially explain olfactory crossmodal correspondences

During the olfactory perception process, our olfactory receptors are thought to recognize specific chemical features. These features may contribute towards explaining our crossmodal perception. The physicochemical features of odors can be extracted using an array of gas sensors, also known as an electronic nose. The present study investigates the role that the physicochemical features of olfactory stimuli play in explaining the nature and origin of olfactory crossmodal correspondences, which is a consistently overlooked aspect of prior work. Here, we answer the question of whether the physicochemical features of odors contribute towards explaining olfactory crossmodal correspondences and by how much. We found a similarity of 49% between the perceptual and the physicochemical spaces of our odors. All of our explored crossmodal correspondences namely, the angularity of shapes, smoothness of textures, perceived pleasantness, pitch, and colors have significant predictors for various physicochemical features, including aspects of intensity and odor quality. While it is generally recognized that olfactory perception is strongly shaped by context, experience, and learning, our findings show that a link, albeit small (6–23%), exists between olfactory crossmodal correspondences and their underlying physicochemical features.

The effects of estradiol levels on crossmodal perception: a study on the sound induced flash illusion in healthy and menstrually related migraine individuals

ObjectiveThe sound-induced flash illusion (SIFI) is a valid paradigm to study multisensorial perception. In the “fission” SIFI, multiple flashes are perceived when observing a single flash paired with two or more beeps. SIFI is largely dependent on visual and acoustic cortex excitability; in migraine, dysfunctional cortical excitability affects SIFI perception. Since estrogen peak occurring during ovulation can increase neuronal excitability, the present study aims to verify whether cortical excitability shifts linked to the menstrual cycle could influence SIFI.MethodsIn a comparative prospective study, we tested the effect of estrogens on crossmodal perception using the SIFI. We recruited 27 females in reproductive age, including 16 healthy and 11 menstrually related migraine females, testing their proneness to SIFI on day 14 (high estradiol) and day 27 (low estradiol) of menstrual cycle.ResultsWomen on day 14 reported less flashes than on day 27 (p = 0.02) in the fission illusion, suggesting a pro-excitatory effect of estradiol on visual cortex excitability during ovulation. Moreover, we confirmed that migraine women perceived less flashes (p = 0.001) than controls, independently from cycle phase. Non-migraineurs women significantly reported more flashes on day 27 than on day 14 (p = 0.04).ConclusionsThis study suggests that estradiol may influence the multisensory perception due to changes of visual cortex excitability, with high estradiol peak leading to increased visual cortical sensitivity during ovulation in non-migraineurs. Visual cortex hyperresponsiveness, here reflected by reduced SIFI, is not influenced by estradiol fluctuations in migraine women, as shown by reduced fission effects on day 14 and 27.