Multimodal Stimuli Research Articles

Multimodal stimuli regulate reproductive behavior and physiology in male túngara frogs

Unlike in terrestrial animals, the boundary between internal (e.g., hormones) and external (e.g., social) stimulation can be blurred for aquatic and amphibious species. When chemicals such as hormones and glandular secretions leach into the water, they can further interact with other signaling systems, creating multimodal stimuli. It is unclear, however, whether water-borne chemical secretions from courting male frogs affect the physiology and behavior of their rivals. In order to address this question we first established non-invasive, continuous sampling methods for simultaneously measuring both hormones and behavior in amphibious species. Then, we examined whether interactions between water-borne chemical secretions and conspecific calls affect reproductive behavior and physiology (testosterone and corticosterone) of courting male túngara frogs. Our results demonstrate that conspecific acoustic stimulation alone increases locomotor activity, decreases latency to call, and increases calling behavior but does not alter the amount of hormones excreted. In response to water containing chemical secretions from rivals, but in the absence of calls from other males, males excrete more testosterone. Interestingly, the combined acoustic and chemical stimulus causes a multiplicative increase in both calling behavior and hormonal excretion. Taken together, our results suggest that a multimodal chemical-acoustic stimulus physiologically primes males for aggressive behavior.

I see how you feel: How the dark triad recognizes emotions

The present study (N = 212; 150 females) examined the relationships between the Dark Triad, empathy, and emotion recognition in a cross-cultural sample. The Dark Triad, especially psychopathy, impeded cognitive empathy, which highlights their indifference towards others’ emotional states. Further, the Dark Triad and primary psychopathy, but not secondary psychopathy, hampered emotion recognition, which was measured using multi-modal stimuli. We discuss how measures of psychopathy assess the construct differently, as well as how the relationship between psychopathy and emotion recognition is complicated. Finally, we found that those high on the Dark Triad spectrum were worse at recognizing the emotions of female actors on the Geneva Emotion Recognition Test.

Assessment tools for social cognition in epilepsy

Social cognition enables us to interact with other individuals and socio-cognitive functions are of importance throughout life. In people with focal and idiopathic epilepsies, previous research has shown that possible impairments in social cognition may cause interpersonal problems and have a negative impact on quality of life. For this reason, the assessment of socio-cognitive functions in individuals with epilepsy might help to identify individuals at risk and to define targets for intervention. Yet, previous tests were limited to static images and text-based material lacking ecological validity. Therefore, we have developed a novel test battery that aims at a comprehensive understanding of social cognition and uses highly ecologic tasks based on dynamic and multimodal stimulus material. Preliminary results that were obtained with the test battery are outlined in this overview.

Sex Differences in Facial and Vocal Attractiveness Among College Students in China.

This study aims to investigate sex differences in ratings for facial attractiveness (FA) and vocal attractiveness (VA). Participants (60 undergraduates in Study 1 and 111 undergraduates in Study 2) rated the attractiveness of computerized face images and voice recordings of men and women. In Study 1, face images and voice recordings were presented separately. Results indicated that men generally rated voice recordings of women more attractive than those of men, whereas women did not show different attractiveness ratings for voices of men vs. women. In Study 2, face images and voice recordings were paired as multimodal stimuli and presented simultaneously. Results indicated that men rated multimodal stimuli of women as more attractive than those of men, whereas women did not differentiate multimodal stimuli of men vs. women. We found that, compared to VA, FA had a stronger influence on participants' overall evaluations. Finally, we tested the difference between “original multimodal stimuli” (OMS) and “non-original multimodal stimuli” (non-OMS) and found the “OMS-facilitating effect.” Taken together, findings indicated some sex differences in FA and VA in the current study, which could be used to interpret behaviors of sexual selection, human mate preferences, and designs and popularization of sex robots.

Male courtship signal modality and female mate preference in the wolf spider Schizocosa ocreata: results of digital multimodal playback studies.

Females must be able to perceive and assess male signals, especially when they occur simultaneously with those of other males. Previous studies show female Schizocosa ocreata wolf spiders display receptivity to isolated visual or vibratory courtship signals, but increased receptivity to multimodal courtship. It is unknown whether this is true when females are presented with a choice between simultaneous multimodal and isolated unimodal male courtship. We used digital playback to present females with a choice simulating simultaneous male courtship in different sensory modes without variation in information content: 1) isolated unimodal visual versus vibratory signals; 2) multimodal versus vibratory signals; and 3) multimodal versus visual signals. When choosing between isolated unimodal signals (visual or vibratory), there were no significant differences in orientation latency and number of orientations, approaches or receptive displays directed to either signal. When given a choice between multimodal versus vibratory-only male courtship signals, females were more likely to orient to the multimodal stimulus, and directed significantly more orients, approaches and receptivity behaviors to the multimodal signal. When presented with a choice between multimodal and visual-only signals, there were significantly more orients and approaches to the multimodal signal, but no significant difference in female receptivity. Results suggest that signal modes are redundant and equivalent in terms of qualitative responses, but when combined, multimodal signals quantitatively enhance detection and/or reception. This study confirms the value of testing preference behavior using a choice paradigm, as female preferences may depend on the context (e.g., environmental context and social context) in which they are presented with male signals.

Environmental heterogeneity alters mate choice behavior for multimodal signals

Animals frequently experience changes in their environment, including diel and seasonal shifts in abiotic and biotic factors. In addition to physiological and morphological changes, animals alter their behavior in response to environmental variation. This study examined the impacts of heterogeneous environments on mating behaviors. We examined both male and female tungara frog phonotactic responses to multimodal (audiovisual) and unimodal (acoustic) stimuli. We altered aspects of the physical environment by changing substrate (terrestrial and aquatic) and ambient light levels. Females demonstrated a similar preference for the audiovisual stimulus regardless of substrate but decreased latency to choose in an aquatic environment. When ambient light levels were increased (relative to darker control), females reversed their preference, avoiding the multimodal stimulus, but the latency to choose was unchanged. Males demonstrated no preference for the multimodal signal on either substrate, but like females, male latency was reduced in an aquatic environment. Different environments carry their own associated costs, including varying levels of predation risk. Increased light levels and an aquatic environment likely carry higher predation risk and therefore should lead to changes in female and male responses. Interestingly, these two environments do not cause uniform changes in female responses. The addition of an aquatic environment led to a reduction in latency, whereas an increase in ambient light levels induced a change in female mate preference. These findings demonstrate the importance of the environment on mating responses to multimodal signals. Responses to conspecific mating signals (e.g., mate preferences) have often been assumed to be static. In this study, we tested responses of both male and female tungara frogs to a multimodal (visual + vocal) advertisement signal under varying environmental conditions. Elevated light levels changed female responses to mating signals. When both sexes were required to swim, rather than walk to a signal, their choices did not change. They did respond faster, however. These results indicate that measuring the strength of sexual selection should be considered as a function of both innate preferences and the environmental conditions experienced by the animals.

Spontaneous colour preferences and colour learning in the fruit-feeding butterfly, Mycalesis mineus

Spontaneous colour preferences have been extensively studied in flower-visiting insects and such preferences exhibited by inexperienced flower-visiting insects are proposed to be adaptive by guiding them to the most rewarding flowers. Thus, spontaneous preferences are hypothesised to reflect the floral reward properties of the habitats in which these insects evolved. However, little is known about colour preferences in non-flower-visiting insects, and the ecological significance of such preferences. We investigated spontaneous colour preferences in the context of feeding and colour learning in the obligate fruit-feeding satyrine butterfly Mycalesis mineus. We report that M. mineus has true colour vision and naive butterflies, both males and females, did not prefer any colour in the absence of olfactory cues in a four-colour test array. Interestingly, females preferred yellow in the presence of food odour, indicating the modulatory effect of olfactory cues on spontaneous colour preferences. Further, when yellow was replaced with orange, female preference shifted to red, demonstrating the influence of colour combinations used in the test. We also report that M. mineus females rapidly learn to associate colour, both the preferred yellow and the non-preferred blue, with fermented banana as reward. Thus, for the first time, we report spontaneous colour preferences and colour learning in a non-flower-visiting butterfly. We also demonstrate that the colour preferences are sexually dimorphic in M. mineus and argue that multimodal stimuli are important for foraging decisions in fruit-feeding butterflies. Spontaneous colour preferences exhibited by naive flower-visiting insects are thought to reflect sensory biases and are hypothesised to be adaptive as they guide insects to rewarding flowers. However, such colour preferences in non-flower-visiting insects and their adaptive significance remain unclear. We demonstrate that Mycalesis mineus, a fruit-feeding butterfly, has true colour vision and exhibits spontaneous colour preferences and these preferences are sexually dimorphic. We also demonstrate that colour preferences are modulated by olfactory cues and are influenced by colour combinations presented in tests. We also report that female butterflies rapidly learn to associate colour with rewards. Our results demonstrate that naive fruit-feeding butterflies rely on both visual and olfactory modalities and depend on spontaneous and learnt behaviours while making foraging decisions.

Methodological contributions of Social Neuroscience for the study of multimodality in translation

This study analyses the interplay of various communication modes that enables emotions to be transmitted efficiently from source (ST) to target text (TT) in audio description (AD) as a multimodal text. It draws on existing experimental designs, including neutral or emotional conditions based on the congruency of stimuli from various modes (images, dialogue semantic content or prosody in a film, together with the semantic content of AD). This article reviews the methodological contribution that Social Neuroscience could make to the study of multimodal translation. To this end, some neurobiological models and studies are quoted regarding multimodal emotional information processing (Brück, Kreifelts, & Wildgruber, 2011), the impact of multimodal emotional processing on subjects’ empathy (Regenbogen et al., 2012) and the dynamics of neural networks involved in human empathy and communication through the presentation of multimodal stimuli (Regenbogen, Habel, & Kellerman, 2013). Finally, an experimental design that focuses on the transfer of feelings and emotions in film AD, which would be suitable for a potential pilot study, is presented.

Multimodal Object Classification Models Inspired by Multisensory Integration in the Brain.

Two multimodal classification models aimed at enhancing object classification through the integration of semantically congruent unimodal stimuli are introduced. The feature-integrating model, inspired by multisensory integration in the subcortical superior colliculus, combines unimodal features which are subsequently classified by a multimodal classifier. The decision-integrating model, inspired by integration in primary cortical areas, classifies unimodal stimuli independently using unimodal classifiers and classifies the combined decisions using a multimodal classifier. The multimodal classifier models are implemented using multilayer perceptrons and multivariate statistical classifiers. Experiments involving the classification of noisy and attenuated auditory and visual representations of ten digits are designed to demonstrate the properties of the multimodal classifiers and to compare the performances of multimodal and unimodal classifiers. The experimental results show that the multimodal classification systems exhibit an important aspect of the “inverse effectiveness principle” by yielding significantly higher classification accuracies when compared with those of the unimodal classifiers. Furthermore, the flexibility offered by the generalized models enables the simulations and evaluations of various combinations of multimodal stimuli and classifiers under varying uncertainty conditions.

Mismatch in receiver responses to multimodal signals in a diurnal gecko

Multimodal signals are used by many animals for intraspecific communication and can provide information about sex identity as well as quality of the signaller. In diurnal geckos of the genus Cnemaspis, chromatic colour patches in males evolved after chemical secretions, providing us with an opportunity to examine the utility of evolving multimodality. We quantified signal components and receiver responses in Cnemaspis mysoriensis to determine the relative importance of chemical and visual traits for intraspecific communication. Digital imagery and spectrophotometry of lizards revealed the presence of two distinct male morphs (yellow-gular and white-gular) and one female form (white-gular). All males, but no females, had yellow eye rims. Characterization of chemical secretions from the ventral precloacal and femoral glands of all lizard forms revealed no differences between male morphs. However, all males differed from females in a few key compounds. We then exposed lizards to only chemical stimuli, only visual stimuli, or both chemical and visual stimuli of conspecific males and females. We found that females were responsive to the chemical stimuli alone as well as the multimodal stimuli of males, whereas males were only responsive to the multimodal stimuli of other males. Neither chemical or visual components of females elicited a response from conspecifics. Thus, while the chemical secretions of males are sufficient for females to elicit a response, multimodal stimuli are necessary for males to respond. Based on variation in signalling traits and receiver responses, we conclude that: (1) chemical secretions signal both sex identity and male quality; (2) eye rim colour encodes information about sex identity; and (3) gular colour in males is probably not a redundant trait, providing some information to males, but not females. We conclude that the secondary evolution of visual signals in C. mysoriensis therefore enhances male–male social interactions and not communication in general.

Reduced resting-state connectivity in areas involved in processing of face-related social cues in female adolescents with atypical anorexia nervosa

Atypical anorexia nervosa (AN) has a high incidence in adolescents and can result in significant morbidity and mortality. Neuroimaging could improve our knowledge regarding the pathogenesis of eating disorders (EDs), however research on adolescents with EDs is limited. To date no neuroimaging studies have been conducted to investigate brain functional connectivity in atypical AN. We investigated resting-state functional connectivity using 3 T MRI in 22 drug-naïve adolescent patients with atypical AN, and 24 healthy controls. Psychological traits related to the ED and depressive symptoms have been assessed using the Eating Disorders Examination Questionnaire (EDE-Q) and the Montgomery–Åsberg Depression Rating Scale self-reported (MADRS-S) respectively. Reduced connectivity was found in patients in brain areas involved in face-processing and social cognition, such as the left putamen, the left occipital fusiform gyrus, and specific cerebellar lobules. The connectivity was, on the other hand, increased in patients compared with controls from the right inferior temporal gyrus to the superior parietal lobule and superior lateral occipital cortex. These areas are involved in multimodal stimuli integration, social rejection and anxiety. Patients scored higher on the EDE-Q and MADRS-S questionnaires, and the MADRS-S correlated with connectivity from the right inferior temporal gyrus to the superior parietal lobule in patients. Our findings point toward a role for an altered development of socio-emotional skills in the pathogenesis of atypical AN. Nonetheless, longitudinal studies will be needed to assess whether these connectivity alterations might be a neural marker of the pathology.

Temporal dynamics of audiovisual affective processing

This study used event-related potentials (ERPs) to investigate the time course of auditory, visual, and audiovisual affective processing. Stimuli consisted of naturalistic silent videos, instrumental music clips, or combination of the two, with valence varied at three levels for each modality and arousal matched across valence conditions. Affective ratings of the unimodal and multimodal stimuli showed evidence of visual dominance, congruency, and negativity dominance effects. ERP results for unimodal presentations revealed valence effects in early components for both modalities, but only for the visual condition in a late positive potential. The ERP results for multimodal presentations showed effects for both visual valence and auditory valence in three components, early N200, P300 and LPP. A modeling analysis of the N200 component suggested its role in the visual dominance effect, which was further supported by a correlation between behavioral visual dominance scores and the early ERP components. Significant congruency comparisons were also found for N200 amplitudes, suggesting that congruency effects may occur early. Consistent differences between negative and positive valence were found for both visual and auditory modalities in the P300 at anterior electrode clusters, suggesting a potential source for the negativity dominance effect observed behaviorally. The separation between negative and positive valence also occurred at LPP for the visual modality. Significant auditory valence modulation was found for the LPP, implying an integration effect in which valence sensitivity of the LPP emerged for the audiovisual condition. These results provide a basis for mapping out the temporal dynamics of audiovisual affective processing.

Modeling Development of Multimodal Emotion Perception Guided by Tactile Dominance and Perceptual Improvement

Humans recognize others’ emotional states such as delight, anger, sorrow, and pleasure through their multimodal expressions. However, it is unclear how this capability of emotion perception is acquired during infancy. This paper presents a neural network model that reproduces the developmental process of emotion perception through an infant–caregiver interaction. This network comprises hierarchically structured restricted Boltzmann machines (RBMs) that receive multimodal expressions from a caregiver (visual, audio, and tactile signals in our current experiment) and learn to estimate her/his emotional states. We hypothesize that emotional categories of multimodal stimuli are acquired in a higher layer in the network owing to two important functions: 1) tactile dominance and 2) perceptual improvement. The former refers to that tactile sensors can detect emotional valence of stimuli such as positive, negative, and zero valence more directly than can other sensors due to characteristics of the nerve systems of the skin. This function was implemented as semisupervised learning in the model. The latter refers to developmental changes in the perceptual acuity, which was replicated by refining the variance parameters of the low-layered RBMs. Experimental results demonstrated that tactile dominance and perceptual improvement have the role of facilitating the differentiation of emotional states of multimodal expressions; however, the influences only appear when both functions are included in the model together. Considering our results from the psychological perspective may help to elucidate the neural and social mechanisms of the development of emotion perception.

Multimodal Signal Testing Reveals Gestural Tapping Behavior in Spotted Reed Frogs.

In the majority of anuran species, acoustic signals are the dominant mode of inter- and intrasexual communication. Male calls are always accompanied by the movement of a more or less conspicuous vocal sac-a potential visual cue. Reed frogs possess a striking vocal sac with a colorful patch of gland tissue clearly visible once the vocal sac is inflated during acoustic signaling. To investigate the visual signal function of vocal sac and gular gland, we presented male Spotted Reed Frogs (Hyperolius puncticulatus) with unimodal and multimodal signal playbacks of conspecific rivals in their natural habitat and recorded their behavioral responses. We found no difference in receiver response to unimodal advertisement call stimuli and to multimodal stimulus presentations of calls combined with visual signals of an artificial vocal sac with or without a gular patch, moving synchronously or asynchronously with the call playback. The inflations of a vocal sac with a colorful gular patch did not alter receiver response and neither increase nor decrease signal salience during male-male communication. Interestingly, males frequently displayed a novel hind and front foot-tapping behavior in response to all playbacks. Comparison of male responses to advertisement and aggressive call playbacks showed that Spotted Reed Frogs approached the sound source less during aggressive call presentations. Tapping behavior was not influenced by either call playback. We suggest that the gestural tapping behavior might act as vibrational signal and discuss its potential signal function in male contests and courtship for females.

Natural Walking in Virtual Reality

Recent technological developments have finally brought virtual reality (VR) out of the laboratory and into the hands of developers and consumers. However, a number of challenges remain. Virtual travel is one of the most common and universal tasks performed inside virtual environments, yet enabling users to navigate virtual environments is not a trivial challenge—especially if the user is walking. In this article, we initially provide an overview of the numerous virtual travel techniques that have been proposed prior to the commercialization of VR. Then we turn to the mode of travel that is the most difficult to facilitate, that is, walking. The challenge of providing users with natural walking experiences in VR can be divided into two separate, albeit related, challenges: (1) enabling unconstrained walking in virtual worlds that are larger than the tracked physical space and (2) providing users with appropriate multisensory stimuli in response to their interaction with the virtual environment. In regard to the first challenge, we present walking techniques falling into three general categories: repositioning systems, locomotion based on proxy gestures, and redirected walking. With respect to multimodal stimuli, we focus on how to provide three types of information: external sensory information (visual, auditory, and cutaneous), internal sensory information (vestibular and kinesthetic/proprioceptive), and efferent information. Finally, we discuss how the different categories of walking techniques compare and discuss the challenges still facing the research community.

An Innovative Information Technology Educational Framework Based on Embodied Cognition and Sensory Marketing

This article describes and implements an innovative framework for information technology (IT) education. The proposed framework creates metaphors for various IT topics using music. The theory of embodied cognition or grounded cognition argues that all aspects of cognition, including decision making, are shaped by aspects of the body. Various theories of neuroscience, the interdisciplinary study of the nervous system, are used to explain how the brain processes the information and multi-modal stimuli generated by the authors' model. The framework proposed in this article can also be considered a form of sensory marketing, which is also based upon embodied cognition, theories of neuroscience and the cognitive significance of metaphors. The model was implemented at the secondary and university levels using both a formative and summative evaluation process. The survey results support the theoretical arguments supplied by many theories of embodied cognition and neuroscience.

Healthy young adults implement distinctive avoidance strategies while walking and circumventing virtual human vs. non-human obstacles in a virtual environment

This study examines how three types of obstacles (cylinder, virtual human and virtual human with footstep sounds) affect circumvention strategies of healthy young adults. Sixteen participants aged 25.2 ± 2.5 years (mean ± 1SD) were tested while walking overground and viewing a virtual room through a helmet mounted display. As participants walked towards a stationary target in the far space, they avoided an obstacle (cylinder or virtual human) approaching either from the right (+40°), left (−40°) or head-on (0°). Obstacle avoidance strategies were characterized using the position and orientation of the head. Repeated mixed model analysis showed smaller minimal distances (p = 0.007) while avoiding virtual humans as compared to cylinders. Footstep sounds added to virtual humans did not modify (p = 0.2) minimal distances compared to when no sound was provided. Onset times of avoidance strategies were similar across conditions (p = 0.06). Results indicate that the nature of the obstacle (human-like vs. non-human object) matters and can modify avoidance strategies. Smaller obstacle clearances in response to virtual humans may reflect the use of a less conservative avoidance strategy, due to a resemblance of obstacles to pedestrians and a recall of strategies used in daily locomotion. The lack of influence of footstep sounds supports the fact that obstacle avoidance primarily relies on visual cues and the principle of ‘inverse effectiveness’ whereby multisensory neurons’ response to multimodal stimuli becomes weaker when the unimodal sensory stimulus (vision) is strong. Present findings should be taken into consideration to optimize the ecological validity of VR-based obstacle avoidance paradigms used in research and rehabilitation.

Computational Modelling of Cerebellum Granule Neuron Temporal Responses for Auditory and Visual Stimuli

Sensorimotor signals from the cerebral cortex modulate the pattern generating metaheuristic capabilities of cerebellum. To better understand the functional integration of multisensory information by the single granule neurons and the role of multimodal information in motor guidance of cerebellum, we have modelled granular layer microcircuit in the cerebellum and analysed the encoding of information during the auditory and visual stimuli. A multi-compartmental granule neuron model comprising of excitatory and inhibitory synapses was used and in vivo like behaviour was modelled with the short and long bursts. The change in intrinsic parameters in the model helped to quantify the effect of spike-time dependent plasticity in the firing of granule neurons. Computer simulations implicate coding correlation of output patterns to temporal excitatory stimuli. We observed the role of induced plasticity and granular layer role in sparse recoding of auditory and visual inputs and the model predict how plasticity mechanisms affect the average amount of information transmitted through the single granule neurons during multimodal stimuli.

An Interactive System for Fine Motor Rehabilitation.

One of the most important aspects in neuromotor rehabilitation is the need of feedback for patients. The rehabilitation system's efficiency relies on the therapist's judgment; the therapist tells the patient whether he/she is performing the exercises correctly. This process may be quite subjective, because it depends on the therapist's personal opinion. On the other hand, recent studies have shown that vibrotactile biofeedback can improve the effectiveness of interaction as it is a very helpful tool in the physiological process of neuromotor rehabilitation. We designed an interactive system focused on rehabilitation of the upper limbs using active markers and image processing, which consists of drawing activities in both augment and virtual reality. The system gives the user a correction through multimodal stimuli feedback (vibrotactile, visual, and sound stimulus) and force measurement to let the patients know if they are not achieving the tasks' goals. The developed system could be used by nursing assistants to better help patients. The purpose of this system was assisting patients with injuries in shoulders, elbows, or wrists, providing an audio-vibrotactile feedback as a factor of correction in the movements of the patient. To examine our system, 11 participants were asked to participate in an experiment where they performed activities focused to strengthen their fine motor movements. Results showed show that patients' fine motor skills improved 10% on average by comparing their error rates throughout the sessions.

Multimodal conceptual knowledge influences lexical retrieval speed: evidence from object-naming and word-reading in healthy adults

Multimodal conceptual knowledge influences lexical retrieval speed: evidence from object-naming and word-reading in healthy adults