Bimodal Signals Research Articles

Learning of bimodal versus unimodal signals in restrained bumble bees.

Similar to animal communication displays, flowers emit complex signals that attract pollinators. Signal complexity could lead to higher cognitive load for pollinators, impairing performance, or might benefit them by facilitating learning, memory and decision making. Here, we evaluated learning and memory in foragers of the bumble bee Bombus impatiens trained to simple (unimodal) versus complex (bimodal) signals under restrained conditions. Use of a proboscis extension response protocol enabled us to control the timing and duration of stimuli presented during absolute and differential learning tasks. Overall, we observed broad variation in performance under the two conditions, with bees trained to compound bimodal signals learning and remembering as well as, better than or more poorly than bees trained to unimodal signals. Interestingly, the outcome of training was affected by the specific colour-odour combination. Among unimodal stimuli, the performance with odour stimuli was higher than with colour stimuli, suggesting that olfactory signals played a more significant role in the compound bimodal condition. This was supported by the fact that after 24 h, most bimodal-treatment bees responded to odour but not visual stimuli. We did not observe differences in latency of response, suggesting that signal composition affected decision accuracy, not speed. We conclude that restrained bumble bee workers exhibit broad variation of responses to bimodal stimuli and that components of the bimodal signal may not be used equivalently. The analysis of bee performance under restrained conditions enables accurate control of the multimodal stimuli provided to individuals and to study the interaction of individual components within a compound.

A dual-mode fluorescent and colorimetric immunoassay based on in situ ascorbic acid-induced signal generation from metal-organic frameworks

A novel dual-mode enzyme-linked immunosorbent assay (ELISA) has been developed based on Fe(III)-containing metal–organic frameworks (Fe-MOFs), which possesses unique properties of an ascorbic acid (AA)-responsive fluorescence emission and an oxidase-like activity inhibition. In this design, bimodal signals, a remarkable emission enhancement and a noticeable color change from the oxidation inhibition of a chromogenic substrate 3,3′,5,5′-tetramethylbenzidine (TMB), are generated from the Fe-MOFs probe with the increase of AA concentration. The change in the fluorescent and colorimetric signal is easily regulated by controlling AA level, which in turn is catalytically adjusted by the alkaline phosphatase (ALP) used in ELISA. Under an optimal condition, a limit of detection (LOD) at 180 pg mL−1 is achieved with the proposed dual-mode ELISA using human prostate specific antigen (PSA) as a model analyte. Furthermore, the mechanism of AA-induced emission enhancement of Fe-MOFs is also investigated systematically. The results show that the enhancement is attributed the AA-induced inhibition of ligand-metal charge transfer (LMCT) from the fluorescent ligand 2-amino-1, 4-benzenedicarboxylic acid (BDC-NH2) to Fe(III). When Fe(III) in Fe-MOF is reduced into Fe(II) by AA, the LMCT is partially inhibited and the fluorescence of BDC-NH2 is consequently recovered, thus resulting in a fluorescence enhancement of the Fe-MOFs.

Multisensory enhancement of burst activity in an insect auditory neuron.

Detecting predators is crucial for survival. In insects, a few sensory interneurons receiving sensory input from a distinct receptive organ extract specific features informing the animal about approaching predators and mediate avoidance behaviors. Although integration of multiple sensory cues relevant to the predator enhances sensitivity and precision, it has not been established whether the sensory interneurons that act as predator detectors integrate multiple modalities of sensory inputs elicited by predators. Using intracellular recording techniques, we found that the cricket auditory neuron AN2, which is sensitive to the ultrasound-like echolocation calls of bats, responds to airflow stimuli transduced by the cercal organ, a mechanoreceptor in the abdomen. AN2 enhanced spike outputs in response to cross-modal stimuli combining sound with airflow, and the linearity of the summation of multisensory integration depended on the magnitude of the evoked response. The enhanced AN2 activity contained bursts, triggering avoidance behavior. Moreover, cross-modal stimuli elicited larger and longer lasting excitatory postsynaptic potentials (EPSP) than unimodal stimuli, which would result from a sublinear summation of EPSPs evoked respectively by sound or airflow. The persistence of EPSPs was correlated with the occurrence and structure of burst activity. Our findings indicate that AN2 integrates bimodal signals and that multisensory integration rather than unimodal stimulation alone more reliably generates bursting activity. NEW & NOTEWORTHY Crickets detect ultrasound with their tympanum and airflow with their cercal organ and process them as alert signals of predators. These sensory signals are integrated by auditory neuron AN2 in the early stages of sensory processing. Multisensory inputs from different sensory channels enhanced excitatory postsynaptic potentials to facilitate burst firing, which could trigger avoidance steering in flying crickets. Our results highlight the cellular basis of multisensory integration in AN2 and possible effects on escape behavior.

Error Weighted Semi-Coupled Hidden Markov Model for Audio-Visual Emotion Recognition

This paper presents an approach to the automatic recognition of human emotions from audio-visual bimodal signals using an error weighted semi-coupled hidden Markov model (EWSC-HMM). The proposed approach combines an SC-HMM with a state-based bimodal alignment strategy and a Bayesian classifier weighting scheme to obtain the optimal emotion recognition result based on audio-visual bimodal fusion. The state-based bimodal alignment strategy in SC-HMM is proposed to align the temporal relation between audio and visual streams. The Bayesian classifier weighting scheme is then adopted to explore the contributions of the SC-HMM-based classifiers for different audio-visual feature pairs in order to obtain the emotion recognition output. For performance evaluation, two databases are considered: the MHMC posed database and the SEMAINE naturalistic database. Experimental results show that the proposed approach not only outperforms other fusion-based bimodal emotion recognition methods for posed expressions but also provides satisfactory results for naturalistic expressions.

Investigating the In-Between: Multisensory Integration of Auditory and Visual Motion Streams

We investigated audiovisual interactions in motion perception by behavioral experiments testing both, the influence of visual stimuli on auditory apparent motion and the influence of auditory stimuli on visual apparent motion perception. A set of loudspeakers with an LED mounted in the middle of each speaker cone was arranged in a semicircle. Apparent motion streams were presented for each modality alone in the unimodal conditions. In the bimodal conditions, stimuli of the second modality were added to fill the temporal and spatial gaps of the sampled trajectory of the reference stream. The participants' task was to observe the quasi-naturalistic stimulus sequences and to perform a standard classification. The addition of stimuli of the second modality indeed facilitated apparent motion perception. Bimodal presentation increased the upper temporal interval up to which the stimuli could be separated in time while still being perceived as continuous motion. We interpret these results as evidence for an ecologically advantageous audiovisual motion integration mechanism which operates beyond the constraints of strict spatiotemporal coincidence. Functional considerations suggest that this mechanism may represent an amodal stage suited for the processing of both unimodal and bimodal signals.

When Correlation Implies Causation in Multisensory Integration

Inferring which signals have a common underlying cause, and hence should be integrated, represents a primary challenge for a perceptual system dealing with multiple sensory inputs [1-3]. This challenge is often referred to as the correspondence problem or causal inference. Previous research has demonstrated that spatiotemporal cues, along with prior knowledge, are exploited by the human brain to solve this problem [4-9]. Here we explore the role of correlation between the fine temporal structure of auditory and visual signals in causal inference. Specifically, we investigated whether correlated signals are inferred to originate from the same distal event and hence are integrated optimally [10]. In a localization task with visual, auditory, and combined audiovisual targets, the improvement in precision for combined relative to unimodal targets was statistically optimal only when audiovisual signals were correlated. This result demonstrates that humans use the similarity in the temporal structure of multisensory signals to solve the correspondence problem, hence inferring causation from correlation.

Audiovisual Interaction in Time Perception

We examined the cross-modal effect of irrelevant sound (or disk) on the perceived visual (or auditory) duration, and how visual and auditory signals are integrated when perceiving the duration. Participants conducted a duration discrimination task with a 2-Interval-Forced-Choice procedure, with one interval containing the standard duration and the other the comparison duration. In study 1, the standard and comparison durations were either in the same modality or with another modality added. The point-of-subjective-equality and threshold were measured from the psychometric functions. Results showed that sound expanded the perceived visual duration at the intermediate durations but there was no effect of disk on the perceived auditory duration. In study 2, bimodal signals were used in both the standard and comparison durations and the Maximum-Likelihood-Estimation (MLE) model was used to predict bimodal performance from the observed unimodal results. The contribution of auditory signals to the bimodal estim...

Load Analysis of Transmitting Gear in Loading Cycles for Earthmoving Vehicle

Fatigue failure of gears carrying heavy loads is the leading cause of gearbox breakdown in the large-scale earthmoving machinery. The fatigue life of gear is mainly determined by loads distribution. Taking the Wheel Loader as an example, this paper introduced an indirect test system of gear load under the loose soil conditions. Then the load data in time, amplitude and frequency field were respectively analyzed. The results show that the indirect measurement method proved workable, and the load data of gear is characterized by bi-modal. Therefore, to decrease statistical errors, mixture model must be adopted to fit these bi-modal signals.

The elicitation of audiovisual steady-state responses: multi-sensory signal congruity and phase effects.

Most ecologically natural sensory inputs are not limited to a single modality. While it is possible to use real ecological materials as experimental stimuli to investigate the neural basis of multi-sensory experience, parametric control of such tokens is limited. By using artificial bimodal stimuli composed of approximations to ecological signals, we aim to observe the interactions between putatively relevant stimulus attributes. Here we use MEG as an electrophysiological tool and employ as a measure the steady-state response (SSR), an experimental paradigm typically applied to unimodal signals. In this experiment we quantify the responses to a bimodal audio-visual signal with different degrees of temporal (phase) congruity, focusing on stimulus properties critical to audiovisual speech. An amplitude modulated auditory signal ('pseudo-speech') is paired with a radius-modulated ellipse ('pseudo-mouth'), with the envelope of low-frequency modulations occurring in phase or at offset phase values across modalities. We observe (i) that it is possible to elicit an SSR to bimodal signals; (ii) that bimodal signals exhibit greater response power than unimodal signals; and (iii) that the SSR power at specific harmonics and sensors differentially reflects the congruity between signal components. Importantly, we argue that effects found at the modulation frequency and second harmonic reflect differential aspects of neural coding of multisensory signals. The experimental paradigm facilitates a quantitative characterization of properties of multi-sensory speech and other bimodal computations.

Reaction time facilitation for horizontally moving auditory-visual stimuli

For moving targets, bimodal facilitation of reaction time has been observed for motion in the depth plane (C. Cappe, G. Thut, B. Romei, & M. M. Murray, 2009), but it is unclear whether analogous RT facilitation is observed for auditory-visual motion stimuli in the horizontal plane, as perception of horizontal motion relies on very different cues. Here we found that bimodal motion cues resulted in significant RT facilitation at threshold level, which could not be explained using an independent decisions model (race model). Bimodal facilitation was observed at suprathreshold levels when the RTs for suprathreshold unimodal stimuli were roughly equated, and significant RT gains were observed for direction-discrimination tasks with abrupt-onset motion stimuli and with motion preceded by a stationary phase. We found no speeded responses for bimodal signals when a motion signal in one modality was paired with a spatially co-localized stationary signal in the other modality, but faster response times could be explained by statistical facilitation when the motion signals traveled in opposite directions. These results strongly suggest that integration of motion cues led to the speeded bimodal responses. Finally, our results highlight the importance of matching the unimodal reaction times to obtain response facilitation for bimodal motion signals in the linear plane.

Colour, size and movement as visual subcomponents in multimodal communication by the frog Allobates femoralis

Multimodal signals convey information that can be detected by receivers using two or more sensory modalities. The information transmitted through secondary channels may be mostly useless, redundant, complementary, or essential. Although each option implies different evolutionary scenarios, the relative role of signal subcomponents in a secondary channel has rarely been tested. Anuran amphibians are known for their extensive use of auditory signals, but growing evidence suggests their common use of multimodal signals where the visual component represents the secondary channel. Territorial males of the nontoxic frog Allobates femoralis use bimodal (acoustic þ visual) signals to recognize conspecific intruders. We used playback experiments to test the relative importance of three visual subcomponents (body coloration, body size and body movement) in a multimodal signal to elicit agonistic reactions in males. The probability of attack was highly related to the movement and size of a dummy frog but very poorly related to dummy’s coloration. The results suggest that any body movement, and not only vocal sac movement, is a nonredundant component of a bimodal signal that probably modulates male aggression. Against our expectations, males readily attacked dummy frogs painted to mimic the cooccurring frog Ameerega trivittata as well as models lacking some of their own pattern’s subcomponents (white lines and yellow patches). Colourful frogs did not use body coloration as a visual cue for conspecific recognition. Both patterns recall the importance of carefully manipulating the availability of visual information in interpreting the origin and evolution of multimodal signals.

Bimodal sensory discrimination is finer than dual single modality discrimination

Here we show that discriminating between different signal modulation rates can be easier when stimuli are presented in two modalities (vision and audition) rather than just one. This was true even when the single modality signal was repeated. This facilitation did not require simultaneous presentations in both modalities and therefore cannot rely on sensory fusion. Signal detection threshold for bimodal signals and double single modality signals were found to be equivalent indicating that the double single modality signals were not intrinsically noisier. The lack of facilitation in double single modality conditions was not due to inaccessibility of the first sample because there is no performance difference when noise was added to either the first or second samples. We propose that the bimodal signal discrimination advantage arises from fluctuations in the magnitude of sensory noise over time and because observers select the most reliable modality on a trial by trial basis. Noise levels within repeated single modality trials are more likely to be similar than those within signals from different modalities. As a consequence, signal selection would be less effective in the former circumstances. Overall, our findings illustrate the advantage of using separate sensory channels to achieve reliable information processing.

Redundant visual signals boost saccade execution

The redundant signal effect (RSE) refers to the fact that human beings react more quickly to a pair of stimuli than to only one stimulus. In previous studies of the RSE in the oculomotor system, bimodal signals have been used as the goal of the saccade. In consistency with studies using manual response times (RTs), saccadic RTs have been shown to be shorter for redundant multimodal stimuli than for single unimodal stimuli. In the present experiments, we extended these findings by demonstrating an RSE in the saccadic system elicited only by unimodal visual stimuli. In addition, we found that shorter saccadic RTs were accompanied by an increased saccadic peak velocity. The present results are of relevance for neurophysiological models of saccade execution, since the boost of saccades was elicited by two visual transients (acting as a "go" signal) that were presented not at the goal of the saccade but at various other locations.

Perceptual exploration and intersensory merging in a visual–haptic Necker cube

Perceiving the three-dimensional structure of an object often involves merging vision and haptics over extended periods of exploration. An interesting feature of this process is that as exploration progresses, new information may require changing how the two sensory signals are treated. We tested whether (1) bimodal interactions during extended observation can adjust to changes in the quality of unimodal sensory signals; and (2) there are specific temporal constraints on these adjustments. When viewing a three-dimensional Necker cube with one eye, participants experience illusory reversals even while they feel the cube with their hands. This surprising property makes the visual–haptic Necker cube an excellent model to investigate bimodal processes when previously consistent bimodal signals change in quality over time or begin to conflict. Our participants reported reversals while they viewed the cube and, at the same time, they either held it with two-finger grips, felt it with while their hands remained stationary, or actively explored it by moving one hand. Consistent with a multisensory approach to three-dimensional form perception, touch had a clear effect on both the number and the duration of illusory percepts. Additionally, when observers alternated between stationary and moving periods during exploration, transitions from stationary to moving-hand haptics played a crucial role in inhibiting illusory reversals. A temporal analysis of the probability of first reversals occurring after different types of motor transition revealed a “vetoing window” initiating approximately 2 s after the transition and lasting at least another 1–2 s. These results suggest that intersensory merging is sensitive to changes in the quality of unimodal signals during exploration, and that adjustments caused by these changes are characterized by specific temporal dynamics.

Redundancy Gain and Coactivation in Bimodal Detection: Evidence for the Preservation of Coactive Processing in Older Adults

Previous investigations of adult age differences in the redundant signals effect suggest that both older and younger adults benefit from the presentation of redundant information. However, age deficits in divided attention may cause older adults to process redundant information in a different manner. In the present experiment, we tested between two competing explanations for the redundant signals effect: separate activation and coactivation. To investigate this issue, we used a bimodal detection task in which the auditory signal was a 1000-Hz tone and the visual signal was an asterisk. Both age groups showed significant violations of Miller's race model inequality, providing evidence for coactivation. These results suggest that, despite age-related deficits in divided attention, the ability to coactivate information from bimodal signals is spared with increased age.

Catching audiovisual mice: predicting the arrival time of auditory-visual motion signals.

We investigated the extent to which auditory and visual motion signals are combined when observers are asked to predict the location of a virtually moving target. In Condition 1, the unimodal and bimodal signals were noisy, but the target object was continuously visible and audible; in Condition 2, the virtually moving object was hidden (invisible and inaudible) for a short period prior to its arrival at the target location. Our main finding was that the facilitation due to simultaneous visual and auditory input is very different for the two conditions. When the target is continuously visible and audible (Condition 1), the bimodal performance is twice as good as the unimodal performances, thus suggesting a very effective integration mechanism. On the other hand, if the object is hidden for a short period (Condition 2) and the task therefore requires the extrapolation of motion speed over a temporal and spatial period, the facilitation due to both sensory inputs is almost absent, and the bimodal performance is limited by the visual performance.

Garner and congruence effects in the speeded classification of bimodal signals.

The role of attention in speeded Garner classification of concurrently presented auditory and visual signals was examined in 4 experiments. Within-trial interference (i.e., congruence effects) occurred regardless of the attentional demands of the task. Between-trials interference (i.e., Garner interference) occurred only under conditions of divided attention when making judgments about auditory signals. Of importance, the data show congruence effects in the absence of Garner interference. Such a pattern has been rarely reported in studies of the classification of purely visual stimuli and contradicts theoretical accounts asserting that the effects share a common locus. The data question the notion that Garner classification reveals fundamental insights about the nature of the perceptual processing of bimodal stimuli.

Garner and congruence effects in the speeded classification of bimodal signals.

Garner and congruence effects in the speeded classification of bimodal signals.

Emphasizing Difficulties in the Detection of Rhythms with Lomb-Scargle Periodograms

The Lomb-Scargle periodogram was introduced in astrophysics to detect sinusoidal signals in noisy unevenly sampled time series. It proved to be a powerful tool in time series analysis and has recently been adapted in biomedical sciences. Its use is motivated by handling non-uniform data which is a common characteristic due to the restricted and irregular observations of, for instance, free-living animals. However, the observational data often contain fractions of non-Gaussian noise or may consist of periodic signals with non-sinusoidal shapes. These properties can make more difficult the interpretation of Lomb-Scargle periodograms and can lead to misleading estimates. In this letter we illustrate these difficulties for noise-free bimodal rhythms and sinusoidal signals with outliers. The examples are aimed to emphasize limitations and to complement the recent discussion on Lomb-Scargle periodograms.

Hydrodynamic Coupling of Lobster Antennule Motion to Oscillatory Water Flow.

removal of the entire lateral flagellum (2, 4). This could mean that the hydrodynamic receptors, still intact, provide partial, but substantial information for tracking. In the two papers that follow, we begin to focus on the lateral flagellum as a detector of hydrodynamic perturbations. Weaver and Atema analyze the hydrodynamic coupling of this organ to imposed water motion and conclude that both stretch and flow receptors could be detecting the observed frequency-dependent pivoting and whipping motions of the flagellum. Guenther and Atema describe the distribution of setae on the flagellum, including those with the potential for detecting relative water flow (Figure 1, right). Chemoreception is not infrequently coincident with mechanoreception. Taste in vertebrates-from fish to humans-is mediated by taste buds that are innervated by both chemoreceptor and mechanoreceptor cells. The two receptor systems obtain information from exactly the same locations-i.e., each single taste budthen carry their information separately to the brain; there the systems project again in close proximity, interact, and form a joint spatial map of the mouth and gut, leading to bimodal output (8). The functional aim is obvious: animals need to sort the tasty items from among the others in the mouth. The flavored eddies of interest to the lobster present similar bimodal signals to its olfactory system, and that coincidence detection then provides superior information for tracking odor plumes. -Jelle Atema August 1998