Crossmodal Research Research Articles

Towards sensory substitution and augmentation: Mapping visual distance to audio and tactile frequency.

Multimodal perception is the predominant means by which individuals experience and interact with the world. However, sensory dysfunction or loss can significantly impede this process. In such cases, cross-modality research offers valuable insight into how we can compensate for these sensory deficits through sensory substitution. Although sight and hearing are both used to estimate the distance to an object (e.g., by visual size and sound volume) and the perception of distance is an important element in navigation and guidance, it is not widely studied in cross-modal research. We investigate the relationship between audio and vibrotactile frequencies (in the ranges 47-2,764 Hz and 10-99 Hz, respectively) and distances uniformly distributed in the range 1-12 m. In our experiments participants mapped the distance (represented by an image of a model at that distance) to a frequency via adjusting a virtual tuning knob. The results revealed that the majority (more than 76%) of participants demonstrated a strong negative monotonic relationship between frequency and distance, across both vibrotactile (represented by a natural log function) and auditory domains (represented by an exponential function). However, a subgroup of participants showed the opposite positive linear relationship between frequency and distance. The strong cross-modal sensory correlation could contribute to the development of assistive robotic technologies and devices to augment human perception. This work provides the fundamental foundation for future assisted HRI applications where a mapping between distance and frequency is needed, for example for people with vision or hearing loss, drivers with loss of focus or response delay, doctors undertaking teleoperation surgery, and users in augmented reality (AR) or virtual reality (VR) environments.

Sonic branding: A narrative review at the intersection of art and science

AbstractThe field of sonic branding/marketing has grown rapidly in recent years, as has commercial interest in more science‐based approaches to the practice, particularly as more brands and agencies look for qualitative and quantitative evidence to support the efficacy of sonic branding. In this research note, we explore the ways in which the design and execution (i.e., the “art”) of sonic branding initiatives can be informed by overlapping approaches and techniques drawn from the sciences, namely psychoacoustics, semiotics, music/auditory cognition, and crossmodal research. We explore whether the rapid growth of generative AI may represent the next major evolution in the design, creation, and assessment of sonic assets, where science and art are used to train AI tools that could one day augment (and potentially disrupt) the work of human sound designers and composers. These developments notwithstanding, it is argued that sonic branding will likely remain as much an art as a science, though basing one's approach on the emerging scientific literature ought to at least tilt the odds of success in the creative's favor.

The untapped potential of causal inference in cross-modal research

Cross-modal effects have recently become a popular topic in building science. However, studies in this area frequently neglect causal inference, leading to a lack of valid causal results. To address this problem, we specifically highlight causality and its importance to cross-modal research. We present three general guidelines, and describe them using toy examples, for appropriately conducting causal cross-modal research. The guidelines originate from the methodological framework for quantitative social science by Lundberg et al. (2021). They are as follows: i) specify the theoretical estimand as the target of causal inference; ii) specify the empirical estimand that is informative for the theoretical estimand based on causal assumptions; iii) select the estimation strategy empirically to estimate the empirical estimand. In light of these guidelines, we discuss some common methodological pitfalls in current research practices that can jeopardize causal inference. Moreover, we offer certain recommendations to avoid such pitfalls. The general objective of this paper is to promote transparent causal cross-modal research by raising the awareness of causal inference in view of appropriate causality-related methodological choices.

BrainStat: A toolbox for brain-wide statistics and multimodal feature associations

Analysis and interpretation of neuroimaging datasets has become a multidisciplinary endeavor, relying not only on statistical methods, but increasingly on associations with respect to other brain-derived features such as gene expression, histological data, and functional as well as cognitive architectures. Here, we introduce BrainStat - a toolbox for (i) univariate and multivariate linear models in volumetric and surface-based brain imaging datasets, and (ii) multidomain feature association of results with respect to spatial maps of post-mortem gene expression and histology, task-based fMRI meta-analysis, as well as resting-state fMRI motifs across several common surface templates. The combination of statistics and feature associations into a turnkey toolbox streamlines analytical processes and accelerates cross-modal research. The toolbox is implemented in both Python and MATLAB, two widely used programming languages in the neuroimaging and neuroinformatics communities. BrainStat is openly available and complemented by an expandable documentation.

Cross-modal association analysis and matching model construction of perceptual attributes of multiple colors and combined tones.

Audio-visual correlation is a common phenomenon in real life. In this article, aiming at analyzing the correlation between multiple colors and combined tones, we comprehensively used experimental methods and technologies such as experimental psychology methods, audio-visual information processing technology, and machine learning algorithms to study the correlation mechanism between the multi-color perceptual attributes and the interval consonance attribute of musical sounds, so as to construct an audio-visual cross-modal matching models. Specifically, in the first, this article constructed the multi-color perceptual attribute dataset through the subjective evaluation experiment, namely "cold/warm," "soft/hard," "transparent/turbid," "far/near," "weak/strong," pleasure, arousal, and dominance; and constructed the interval consonance attribute dataset based on calculating the audio objective parameters. Secondly, a subjective evaluation experiment of cross-modal matching was designed and carried out for analyzing the audio-visual correlation, so as to obtain the cross-modal matched and mismatched data between the audio-visual perceptual attributes. On this basis, through visual processing and correlation analysis of the matched and mismatched data, this article proved that there is a certain correlation between multicolor and combined tones from the perspective of perceptual attributes. Finally, this article used linear and non-linear machine learning algorithms to construct audio-visual cross-modal matching models, so as to realize the mutual prediction between the audio-visual perceptual attributes, and the highest prediction accuracy is up to 79.1%. The contributions of our research are: (1) The cross-modal matched and mismatched dataset can provide basic data support for audio-visual cross-modal research; (2) The constructed audio-visual cross-modal matching models can provide a theoretical basis for audio-visual interaction technology; (3) In addition, the research method of audio-visual cross-modal matching proposed in this article can provide new research ideas for related research.

Understanding across the senses: cross-modal studies of cognition in cetaceans.

Cross-modal approaches to the study of sensory perception, social recognition, cognition, and mental representation have proved fruitful in humans as well as in a variety of other species including toothed whales in revealing equivalencies that suggest that different sensory stimuli associated with objects or individuals may effectively evoke mental representations that are, respectively, object based or individual based. Building on established findings of structural equivalence in the form of spontaneous recognition of complex shapes across the modalities of echolocation and vision and behavior favoring identity echoic-visual cross-modal relationships over associative echoic-visual cross-modal relationships, examinations of transitive inference equivalencies from initially learned associations of visual and acoustic stimuli, and recent work examining spontaneous cross-modal social recognition of individual identity across acoustic and gustatory chemical modalities (i.e., the equivalence relationships among an individual's characteristics), we examine the history, utility and implications for cross-modal research in cetacean cognition. Drawing from research findings on bottlenose dolphins and beluga whales as well as other species we suggest future directions for cetacean cross-modal research to further illuminate understanding how structural and individual sensory equivalencies lead to object-centered and individual-centered mental representations, as well as to explore the potential for practical applications related to cetacean conservation.

AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos With Deep Learning

In movie productions, the Foley Artist is responsible for creating an overlay soundtrack that helps the movie come alive for the audience. This requires the artist to first identify the sounds that will enhance the experience for the listener thereby reinforcing the Directors's intention for a given scene. In this paper, we present AutoFoley, a fully-automated deep learning tool that can be used to synthesize a representative audio track for videos. AutoFoley can be used in the applications where there is either no corresponding audio file associated with the video or in cases where there is a need to identify critical scenarios and provide a synthesized, reinforced soundtrack. An important performance criterion of the synthesized soundtrack is to be time-synchronized with the input video, which provides for a realistic and believable portrayal of the synthesized sound. Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as inter-frame relations in fast moving video clips by incorporating an interpolation technique and Temporal Relationship Networks (TRN). We employ a robust multi-scale Recurrent Neural Network (RNN) associated with a Convolutional Neural Network (CNN) for a better understanding of the intricate input-to-output associations over time. To evaluate AutoFoley, we create and introduce a large scale audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. Our experiments show that the synthesized sounds are realistically portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley show over 73% of the test subjects considered the generated soundtrack as original, which is a noteworthy improvement in cross-modal research in sound synthesis.

Research on the Correlation Between the Timbre Attributes of Musical Sound and Visual Color

This article investigated the timbre of musical sound and visual color; comprehensively used experimental psychology, audio information processing and other experimental methods and technical means; and conducted research on the relationship between timbre attributes and visual color. First, we designed and implemented a subjective perception experiment based on audiovisual cross-modal timbre-color correlation. Through the statistical analysis of the experimental data, we obtained a complete timbre-color cross-modal correlation dataset. Second, through the visual processing and correlation analysis of the timbre-color correlation data, it is proven that there is a certain correlation between the timbre dimension and the color dimension. On this basis, we used three algorithms to construct a timbre-color correlation model, namely, multiple linear regression, BP neural network and SVR, and we verified the accuracy of the three models. The timbre-color correlation dataset constructed in this paper can provide basic data support for audiovisual cross-modal research. The timbre-color correlation model constructed in this paper can provide a theoretical basis for cross-modal audiovisual applications. In addition, the timbre-color cross-modality research method in this paper can provide new research ideas for audio-visual cross-modality research.

Across the Unseen Sea and Dinner for Deep Surface Divers

Dinner for Deep Surface Divers uses food as a poetic medium, where eating is depicted as a highly sensual act, while exploring the role of the senses and embodiment in informing our vision. Across the Unseen Sea captures the essence of a multi-sensory banquet that I created and which took place in London, in 2013. I based the event on William Morris’s diary entries from his journey to Iceland, in 1871/73 and “translated” these into a multi-sensory immersive performance. The project was developed in collaboration with Charles Michel (cook and researcher at Charles Spence’s Crossmodal Research laboratory, Oxford), as well as a team of dedicated collaborators from various backgrounds and with different professional expertise (set, sound, scent designers, actors, food historians etc.). The film was created with the intention of communicating the subjective, multi-sensory experience of one of the guests, a writer, by audio-visual.

The Perfect Meal: The Multisensory Science of Food and Dining

The central idea behind this book is deceptively simple: that our perception of flavour is multisensory. Consequently, to serve up great tasting food is not simply a matter of tickling the taste buds but of engaging a whole range of sensory experiences. Drawing on recent innovative research carried out by Spence’s team at the Crossmodal Research Laboratory at Oxford University, the authors argue that we must go beyond four or five individual taste sensations (which respond to particular sensory nodes on the tongue) to focus on the interaction of multisensory experiences as it is here that flavour emerges. They show how a host of sensory elements ranging from the colour of the plate to the frequency of concurrent sounds all affect the ways we perceive flavour.

On the ‘visual’ in ‘audio-visual integration’: a hypothesis concerning visual pathways

Crossmodal interaction conferring enhancement in sensory processing is nowadays widely accepted. Such benefit is often exemplified by neural response amplification reported in physiological studies conducted with animals, which parallel behavioural demonstrations of sound-driven improvement in visual tasks in humans. Yet, a good deal of controversy still surrounds the nature and interpretation of these human psychophysical studies. Here, we consider the interpretation of crossmodal enhancement findings under the light of the functional as well as anatomical specialization of magno- and parvocellular visual pathways, whose paramount relevance has been well established in visual research but often overlooked in crossmodal research. We contend that a more explicit consideration of this important visual division may resolve some current controversies and help optimize the design of future crossmodal research.

Noise and Correlations in Parallel Perceptual Decision Making

Perceptual decisions involve the accumulation of sensory evidence over time, a process that is corrupted by noise [1]. Here, we extend the decision-making framework to crossmodal research [2, 3] and the parallel processing of two distinct signals presented to different sensory modalities like vision and audition. Contrary to the widespread view that multisensory signals are integrated prior to a single decision [4-10], we show that evidence is accumulated for each signal separately and that consequent decisions are flexibly coupled by logical operations. We find that the strong correlation of response latencies from trial to trial is critical to explain the short latencies of multisensory decisions. Most critically, we show that increased noise in multisensory decisions is needed to explain the mean and the variability of response latencies. Precise knowledge of these key factors is fundamental for the study and understanding of parallel decision processes with multisensory signals.

Noise and Correlations in Parallel Perceptual Decision Making

Summary Perceptual decisions involve the accumulation of sensory evidence over time, a process that is corrupted by noise [1]. Here, we extend the decision-making framework to crossmodal research [2, 3] and the parallel processing of two distinct signals presented to different sensory modalities like vision and audition. Contrary to the widespread view that multisensory signals are integrated prior to a single decision [4–10], we show that evidence is accumulated for each signal separately and that consequent decisions are flexibly coupled by logical operations. We find that the strong correlation of response latencies from trial to trial is critical to explain the short latencies of multisensory decisions. Most critically, we show that increased noise in multisensory decisions is needed to explain the mean and the variability of response latencies. Precise knowledge of these key factors is fundamental for the study and understanding of parallel decision processes with multisensory signals.

Sign languages: Contribution to neurolinguistics from cross-modal research

Using sign language research as an example, we argue that both the cross-linguistic descriptive approach to data, advocated by Evans and Levinson (2009), as well as abstract (‘formal’) analyses are necessary steps towards the development of “neurolinguistic primitives” for investigating how human languages are instantiated in the brain.

Exploring the Role of Low Level Visual Processing in Letter-Speech Sound Integration: A Visual MMN Study.

In contrast with for example audiovisual speech, the relation between visual and auditory properties of letters and speech sounds is artificial and learned only by explicit instruction. The arbitrariness of the audiovisual link together with the widespread usage of letter–speech sound pairs in alphabetic languages makes those audiovisual objects a unique subject for crossmodal research. Brain imaging evidence has indicated that heteromodal areas in superior temporal, as well as modality-specific auditory cortex are involved in letter–speech sound processing. The role of low level visual areas, however, remains unclear. In this study the visual counterpart of the auditory mismatch negativity (MMN) is used to investigate the influences of speech sounds on letter processing. Letter and non-letter deviants were infrequently presented in a train of standard letters, either in isolation or simultaneously with speech sounds. Although previous findings showed that letters systematically modulate speech sound processing (reflected by auditory MMN amplitude modulation), the reverse does not seem to hold: our results did not show evidence for an automatic influence of speech sounds on letter processing (no visual MMN amplitude modulation). This apparent asymmetric recruitment of low level sensory cortices during letter–speech sound processing, contrasts with the symmetric involvement of these cortices in audiovisual speech processing, and is possibly due to the arbitrary nature of the link between letters and speech sounds.

Modality-specific auditory and visual temporal processing deficits.

We studied the attentional blink (AB) and the repetition blindness (RB) effects using an audiovisual presentation procedure designed to overcome several potential methodological confounds in previous cross-modal research. In Experiment 1, two target digits were embedded amongst letter distractors in two concurrent streams (one visual and the other auditory) presented from the same spatial location. Targets appeared in either modality unpredictably at different temporal lags, and the participants' task was to recall the digits at the end of the trial. We evaluated both AB and RB for pairs of targets presented in either the same or different modalities. Under these conditions both AB and RB were observed in vision, AB but not RB was observed in audition, and there was no evidence o

Inhibition of return is supramodal: a demonstration between all possible pairings of vision, touch, and audition.

Inhibition of return' (IOR) refers to the delayed detection often found for targets at the same location as a preceding event. We examined whether IOR reflects a truly supramodal phenomenon, in an experiment designed to avoid criticisms of previous crossmodal research. We presented a random sequence of visual, tactile, and auditory targets to either the left or right of central fixation, and tested for IOR between targets in all three modalities when presented successively to the same versus different side. Speeded detection for targets in all three modalities was indeed slower if the preceding target had been presented from the same position, regardless of the modality, of this preceding target. These results demonstrate for the first time that IOR is truly supramodal.

A multimedia HyperCard stack for the computerized investigation of chromesthetic associations

Most researchers investigating the perceptual similarities between visual and auditory stimuli have used different presentation methods for the two stimulus modalities. In contrast, a HyperCard stack has been developed which provides an integrated, fully computerized method for examining the chromesthetic associations of auditory stimuli (both pure tones and musical excerpts) and visual stimuli (48 colored rectangles). Advantages of the stack include control over the presentation of stimuli, control for order effects, a user-friendly interface, limited opportunities for demand characteristics or procedural errors, and simplified data compilation. The multimedia capabilities of HyperCard make it an excellent platform for cross-modal research.