Signals In Multiple Modalities Research Articles

Does the addition of a new signalling trait enhance receiver responses in diurnal geckos?

Animal signals in multiple modalities expands the opportunity for effective communication. Among diurnal geckos of the genus Cnemaspis, chemical signalling traits preceded the evolution of visual traits. Males of all species possess chemical secreting ventral glands, but only in some species, males also express yellow gular patches. This difference in the expression of unimodal or multimodal signalling traits between closely related species provided us with an opportunity to understand the use of multimodal signals for communication. We studied receiver responses in Cnemaspis indica, a sexually monochromatic species, and in C. littoralis, a species where males possess yellow gulars. We performed behavioural trials where individuals of each species were exposed to only chemical stimuli, only visual stimuli, or both chemical and visual stimuli simultaneously from male and female conspecifics. Our results show that only chemical stimuli were necessary and sufficient to elicit responses in males and females of C. indica as well as in females of C. littoralis. However, males of the dimorphic C. littoralis required the multimodal stimulus to elicit movement-based responses. Our results suggest that the evolution of colour traits in diurnal geckos is associated with a partial shift in some receiver responses toward multimodal communication, with no addition to the behavioural repertoire.

Independent Decision Path Fusion for Bimodal Asynchronous Brain–Computer Interface to Discriminate Multiclass Mental States

With the increasing development of brain imaging and sensing technologies, a wide variety of medical signals in multiple modalities have facilitated a better understanding of mental health. The brain-computer interface (BCI) is a technology capable of detecting mental states automatically by employing various brain sensing and machine learning methods. This contributes to efforts involving neurological disease management and restoration of cognitive function. In this study, we present an independent decision path fusion (IDPF) method by developing a bimodal asynchronous BCI based on electroencephalographs (EEGs) and functional near-infrared spectroscopy (fNIRS) to discriminate multiple mental states. The proposed IDPF method generates several independent decision paths, each of which is capable of analyzing cerebral information with respect to a specific aspect, thus interpreting the brain state from multiple points of view. Moreover, in one particular decision path for the fNIRS analysis of the IDPF method, we modified the EEG-based common spatial pattern (CSP) algorithm according to the characteristics of fNIRS. The results show that the modified common spatial pattern (MCSP) significantly outperforms CSP in fNIRS-based BCIs. Through validation on an open-access EEG-fNIRS dataset and comparison with recent studies, we found that our IDPF method achieves a high accuracy of 70.32% for a four-class classification problem (left hand motor imagery, right hand motor imagery, mental arithmetic, and resting state).

Local preference encoded by complex signaling: mechanisms of mate preference in the red-eyed treefrog (Agalychnis callidryas)

The Central American red-eyed treefrog (Agalychnis callidryas) is a nocturnal/crepuscular frog that exhibits high population-level phenotypic variation in a suite of traits generally important in anuran communication. Females of this species demonstrate a preference for local over non-local males, although the contribution of different communication modes to mate choice remains untested. We performed eight sets of experiments at La Selva Biological Station (Costa Rica) to test the relative roles of male advertisement call and flank-stripe pattern on female mate choice. For each trial, we used pre-recorded calls for acoustic signals and/or 3D-printed, hand-painted models for visual stimulus. Females selected conspecifics over heterospecifics when presented with either acoustic or visual stimuli only, indicating that both evolve as species-specific signals used for mate discrimination. Bayesian modeling showed that females chose local males over non-locals based on unimodal; responses to multimodal signals varied. Female mate choice in this species is likely nuanced and subject to the constraints of the local environment in which animals are signaling. Our data show that at least two traits that vary among populations are important for mate selection. Evolution shapes animal communication in diverse ways to accommodate complicated signaling contexts, including environmental noise, the presence of eavesdroppers, and changing landscapes. Among species that use complex signals or signals in multiple modalities, understanding the role of signaling traits in reproductive behavior and their concomitant shifts between populations can shed light on lineage divergence, speciation, and the forces shaping communication. We used choice tests with Agalychnis callidryas to determine which traits lead to local-male mate preference. We found that both call and stripe are each sufficient to allow this nocturnal frog to choose mates in absence of the other stimulus. Multimodal signals were marginally better than call—but not color—alone. This may be a result of statistical noise or preliminary evidence of asymmetric use of traits across a complex landscape. These results suggest that call and color may encode both redundant information and multiple messages to receivers in this species: either is sufficient to elicit mate choice (redundant), but together, they enhance local mate preference (multiple messages). Further tests on how other communication modalities inform behavior will add to our understanding of complex signaling and lineage divergence among differentiated populations in this species.

Human emotion recognition using deep belief network architecture

Recently, deep learning methodologies have become popular to analyse physiological signals in multiple modalities via hierarchical architectures for human emotion recognition. In most of the state-of-the-arts of human emotion recognition, deep learning for emotion classification was used. However, deep learning is mostly effective for deep feature extraction. Therefore, in this research, we applied unsupervised deep belief network (DBN) for depth level feature extraction from fused observations of Electro-Dermal Activity (EDA), Photoplethysmogram (PPG) and Zygomaticus Electromyography (zEMG) sensors signals. Afterwards, the DBN produced features are combined with statistical features of EDA, PPG and zEMG to prepare a feature-fusion vector. The prepared feature vector is then used to classify five basic emotions namely Happy, Relaxed, Disgust, Sad and Neutral. As the emotion classes are not linearly separable from the feature-fusion vector, the Fine Gaussian Support Vector Machine (FGSVM) is used with radial basis function kernel for non-linear classification of human emotions. Our experiments on a public multimodal physiological signal dataset show that the DBN, and FGSVM based model significantly increases the accuracy of emotion recognition rate as compared to the existing state-of-the-art emotion classification techniques.

The wolf spider Pardosa milvina detects predator threat level using only vibratory cues

Predators may inadvertently signal their presence and threat level by way of signals in multiple modalities. We used a spider, Pardosa milvina, known to respond adaptively to chemotactile predator cues (i.e., silk, faeces and other excreta) to evaluate whether it could also discriminate predation risk from isolated vibratory cues. Vibrations from its prey, conspecifics, and predators (Tigrosa helluo and Scarites quadriceps) were recorded and played back to Pardosa. In addition, we recorded predator vibrations with and without access to chemotactile cues from Pardosa, indicating the presence of prey. Pardosa did not appear to discriminate between vibrations from prey or conspecifics, but the response to predators depended on the presence of cues from Pardosa. Vibrations from predators with access to chemotactile cues from prey induced reductions in Pardosa activity. Predator cues typically occur in multiple modalities, but prey are capable of imperfectly evaluating predation risk using a limited subset of information.

A game theoretic approach to multimodal communication

Over the last few decades the animal communica- tion community has become increasingly aware that much communication occurs using multiple signals in multiple modalities. The majority of this work has been empiri- cal, with less theoretical work on the advantages conferred by such communication. In the present paper, we ask: Why should animals communicate with multiple signals in multiple modalities? To tackle this question we use game theoretic techniques, and highlight developments in the economic signaling literature that might offer insight into biological problems. We start by establishing a signaling game, and investigate signal honesty under two prevailing paradigms of honest communication - costly signaling and cheap talk. In both paradigms, without further constraint, it is simple to show that anything that can be achieved with multiple signals can be achieved with one. We go on to investigate different sets of possible constraints that may make multiple signals and multimodal signals in particular more likely to evolve. We suggest that constraints on cost functions and bandwidths, orthogonal noise across modali- ties, strategically distinct modes, multiple qualities, multiple signalers, and multiple audiences, all provide biologically plausible scenarios that theoretically favor multiple and multimodal signaling.

An introduction to multimodal communication

Though it has long been known that animal communication is complex, recent years have seen growing interest in understanding the extent to which animals give multicomponent signals in multiple modalities, and how the different types of information extracted by receivers are interpreted and integrated in animal decision-making. This interest has culminated in the production of the present special issue on multimodal communication, which features both theoretical and empirical studies from leading researchers in the field. Reviews, comparative analyses, and species-specific empirical studies include manuscripts on taxa as diverse as spiders, primates, birds, lizards, frogs, and humans. The present manuscript serves as both an introduction to this special issue, as well as an introduction to multimodal communication more generally. We discuss the history of the study of complexity in animal communication, issues relating to defining and classifying multimodal signals, and particular issues to consider with multimodal (as opposed to multicomponent unimodal) communication. We go on to discuss the current state of the field, and outline the contributions contained within the issue. We finish by discussing future avenues for research, in particular emphasizing that ‘multimodal’ is more than just ‘bimodal’, and that more integrative frameworks are needed that incorporate more elements of efficacy, such as receiver sensory ecology and the environment.

Signaling in multiple modalities in male rhesus macaques: sex skin coloration and barks in relation to androgen levels, social status, and mating behavior.

The past decade has seen an increasing shift in animal communication towards more studies that incorporate aspects of signaling in multiple modalities. Although nonhuman primates are an excellent group for studying the extent to which different aspects of condition may be signaled in different modalities, and how such information may be integrated during mate choice, very few studies of primate species have incorporated such analyses. Here, we present data from free-ranging male rhesus macaques on sex skin coloration (modeled to receiver perception), bark vocal signals, androgen levels, morphometric variables, dominance status, and female mate choice. We show that, consistent with data on females, most intra- and interindividual variation in sex skin appearance occurs in luminance rather than color. Sex skin luminance was significantly correlated across different skin regions. Sex skin luminance did not correlate with the majority of bark parameters, suggesting the potential for the two signals to convey different information. Sex skin appearance was not related to androgen levels although we found some evidence for links between androgen levels and bark parameters, several of which were also related to morphometric variables. We found no evidence that either signal was related to male dominance rank or used in female mate choice, though more direct measures of female proceptive behavior are needed. Overall, the function of male sex skin coloration in this species remains unclear. Our study is among the first nonhuman primate studies to incorporate measurements of multiple signals in multiple modalities, and we encourage other authors to incorporate such analyses into their work.

PSTS response to gaze reflects wider role in processing ostensive signals in multiple modalities

pSTS response to gaze reflects wider role in processing ostensive signals in multiple modalities

Complex courtship displays facilitate male reproductive success and plasticity in signaling across variable environments

Abstract Effective signal transmission is essential for communication. In environments where signal transmission is highly variable, signalers may utilize complex signals, which incorporate multiple components and modalities, to maintain effective communication. Male Rabidosa rabida wolf spiders produce complex courtship signals, consisting of both visual and seismic components. We test the hypothesis that the complex signaling of R. rabida contributes to male reproductive success in variable signaling environments. We first examine the condition-dependence of foreleg ornamentation (a presumed visual signal) and seismic signal components and find that both may provide potentially redundant information on foraging history. Next, we assessed reproductive success across manipulated signaling environments that varied in the effectiveness of visual and/or seismic signal transmission. In environments where only one signal could be successfully transmitted (e.g., visual or seismic), pairs were still able to successfully copulate. Additionally, we found that males altered their courtship display depending on the current signaling environment. Specifically, males reduced their use of a visual display component in signaling environments where visual signal transmission was ablated. Incorporating signals in multiple modalities not only enables R. rabida males to maintain copulation success across variable signaling environments, but it also enables males to adjust their composite courtship display to current signaling conditions.