Concurrent Sounds Research Articles

The Subcomponents of Informational Masking: Evidence from Behavioral and Neural Imaging Studies

The processing of a target speech under noisy environment is influenced by the presence of concurrent sounds, this phenomenon is referred to as auditory masking. Previous research showed that there are two kinds of masking, energetic masking and informational masking. Energetic masking originated from auditory periphery, where the masking sound overlaps with the target sound in time and frequency, resulting in an incomplete representation of the latter. On the other hand, informational masking is believed to occur in the central auditory system. However, for a long time, the concept of informational masking has been used as a "suitcase" or "catchall" word. All the influencing effect that can not be explained by energetic masking is attributed to informational masking, though its underlying mechanism is still largely unknown. In this article, a new conceptual framework of informational masking is proposed. In this framework, informational masking is divided into two subcomponents, perceptual informational masking and cognitive informational masking. Perceptual masking is caused by the competition between the sound stream/object, while cognitive informational masking is caused by the higher cognitive information included in the masking sounds, which could be automatically processed by the auditory or linguistic system. Psychophysical and neural imaging studies are proposed to test the above hypothesis. In behavioral studies, the perceived spatial separation between the target and the masker, the intelligibility and the voice number in the masking sound, as well as the perceptual similarity between the target and masker streams are manipulated to demonstrate that perceptual masking and cognitive masking could be double-dissociated. In the neural imaging study, fMRI technique is used to show that perceptual and cognitive informational masking may have different neural mechanism.

Sound level measurements using smartphone "apps": useful or inaccurate?

Many recreational activities are accompanied by loud concurrent sounds and decisions regarding the hearing hazards associated with these activities depend on accurate sound measurements. Sound level meters (SLMs) are designed for this purpose, but these are technical instruments that are not typically available in recreational settings and require training to use properly. Mobile technology has made such sound level measurements more feasible for even inexperienced users. Here, we assessed the accuracy of sound level measurements made using five mobile phone applications or "apps" on an Apple iPhone 4S, one of the most widely used mobile phones. Accuracy was assessed by comparing application-based measurements to measurements made using a calibrated SLM. Whereas most apps erred by reporting higher sound levels, one application measured levels within 5 dB of a calibrated SLM across all frequencies tested.

Intelligibility of interrupted speech in listeners of different age and hearing status

In most real-world communicative environments, speech signals are fragmented and incomplete due to masking by other concurrent sounds. Experiments in the perception of gated and time-compressed speech provide a useful approach for systematically investigating factors involved in the perception of temporally fragmented speech. In the current work, the intelligibility of spoken sentences was measured following periodic signal interruption at various square-wave gating rates with time compression applied by either omitting or doubling silent intervals during gated-off times. Across interruption rates (0.5–16 Hz), speech perception of younger and older normal-hearing and older hearing-impaired adults was similar for slow interruption rates, but differed substantially for the faster rates. This rate-dependent variation was consistently found for different interruption methods and conditions. Importantly, speech perception at fast interruption rates correlated strongly with pure-tone thresholds and performance on speech-in-noise tests, while it did not correlate with results from tests of working memory or spectro-temporal pattern discrimination. These findings suggest that different perceptual processes are involved in the perception of interrupted speech at slow and at faster interruption rates, and have implications for developing diagnostic tests applicable to real-world listening environments. [Work supported by NIH.]

The Influence of Lifelong Musicianship on Neurophysiological Measures of Concurrent Sound Segregation

The ability to separate concurrent sounds based on periodicity cues is critical for parsing complex auditory scenes. This ability is enhanced in young adult musicians and reduced in older adults. Here, we investigated the impact of lifelong musicianship on concurrent sound segregation and perception using scalp-recorded ERPs. Older and younger musicians and nonmusicians were presented with periodic harmonic complexes where the second harmonic could be tuned or mistuned by 1-16% of its original value. The likelihood of perceiving two simultaneous sounds increased with mistuning, and musicians, both older and younger, were more likely to detect and report hearing two sounds when the second harmonic was mistuned at or above 2%. The perception of a mistuned harmonic as a separate sound was paralleled by an object-related negativity that was larger and earlier in younger musicians compared with the other three groups. When listeners made a judgment about the harmonic stimuli, the perception of the mistuned harmonic as a separate sound was paralleled by a positive wave at about 400 msec poststimulus (P400), which was enhanced in both older and younger musicians. These findings suggest attention-dependent processing of a mistuned harmonic is enhanced in older musicians and provides further evidence that age-related decline in hearing abilities are mitigated by musical training.

Weighting of spatial and spectro-temporal cues for auditory scene analysis by human listeners.

The auditory system creates a neuronal representation of the acoustic world based on spectral and temporal cues present at the listener's ears, including cues that potentially signal the locations of sounds. Discrimination of concurrent sounds from multiple sources is especially challenging. The current study is part of an effort to better understand the neuronal mechanisms governing this process, which has been termed “auditory scene analysis”. In particular, we are interested in spatial release from masking by which spatial cues can segregate signals from other competing sounds, thereby overcoming the tendency of overlapping spectra and/or common temporal envelopes to fuse signals with maskers. We studied detection of pulsed tones in free-field conditions in the presence of concurrent multi-tone non-speech maskers. In “energetic” masking conditions, in which the frequencies of maskers fell within the ±1/3-octave band containing the signal, spatial release from masking at low frequencies (∼600 Hz) was found to be about 10 dB. In contrast, negligible spatial release from energetic masking was seen at high frequencies (∼4000 Hz). We observed robust spatial release from masking in broadband “informational” masking conditions, in which listeners could confuse signal with masker even though there was no spectral overlap. Substantial spatial release was observed in conditions in which the onsets of the signal and all masker components were synchronized, and spatial release was even greater under asynchronous conditions. Spatial cues limited to high frequencies (>1500 Hz), which could have included interaural level differences and the better-ear effect, produced only limited improvement in signal detection. Substantially greater improvement was seen for low-frequency sounds, for which interaural time differences are the dominant spatial cue.

Effects of sex and age on auditory spatial scene analysis

Recently, it has been demonstrated that men outperform women in spatial analysis of complex auditory scenes (Zündorf et al., 2011). The present study investigated the relation between the effects of ageing and sex on the spatial segregation of concurrent sounds in younger and middle-aged adults. The experimental design allowed simultaneous presentation of target and distractor sound sources at different locations. The resulting spatial "pulling" effect (that is, the bias of target localization toward that of the distractor) was used as a measure of performance. The pulling effect was stronger in middle-aged than younger subjects, and female than male subjects. This indicates lower performance of the middle-aged women in the sensory and attentional mechanisms extracting spatial information about the acoustic event of interest from the auditory scene than both younger and male subjects. Moreover, age-specific differences were most prominent for conditions with targets in right hemispace and distractors in left hemispace, suggesting bilateral asymmetries underlying the effect of ageing.

Beeinflussung objektbildender Leistungen des Hörsinns durch multimodale Interaktion

The task of assigning concurrent sounds to different auditory objects is known to depend on temporal and spectral cues. When tones of high and low frequencies are presented in alternation, they can be perceived as a single (integrated) melody, or as two parallel (segregated) melodic lines, according to the presentation rate and frequency distance between the sounds. At an intermediate distance or stimulation rate, the percept is ambiguous and alternates between segregated and integrated. This work studied whether an ambiguous sound organization could be modulated towards a robust integrated or a segregated percept by the synchronous presentation of visual cues. Two interleaved sets of sounds, one high frequency and one low frequency set were presented with concurrent visual stimuli, synchronized to either a within-set frequency pattern or to the across-set intensity pattern. Elicitation of the mismatch negativity (MMN) component of event-related brain potentials served as indices for the segregated organization, when no task was performed with the sounds. As a result, MMN was elicited only when the visual pattern promoted the segregation of the sounds. By spatial analysis of the distribution of electromagnetic potentials, four separated neuronal sources underlying the obtained MMN response were identified. One pair was located bilaterally in temporal cortical structures and another pair in occipital areas, representing the auditory and visual origin of the MMN response, evoked by inverted triplets as used in this study. Thus, the results demonstrate cross-modal effects of visual information on auditory object perception.

Orienting attention to sound object representations attenuates change deafness.

According to the object-based account of attention, multiple objects coexist in short-term memory (STM), and we can selectively attend to a particular object of interest. Although there is evidence that attention can be directed to visual object representations, the assumption that attention can be oriented to sound object representations has yet to be validated. Here, we used a delayed match-to-sample task to examine whether orienting attention to sound object representations influences change detection within auditory scenes consisting of 3 concurrent sounds, each occurring at a different location. On some trials, the 2 scenes were identical; in the remaining trials, the locations of 2 sounds were switched. In a control experiment, we first identified auditory scenes, in which the 3 sounds were unambiguously segregated, for the subsequent experiments. In 2 experiments, we showed that orienting attention to a sound object representation during memory retention (via a retro-cue) enhanced performance relative to uncued trials, up to 4 s of memory retention. Our study shows that complex auditory scenes composed of cooccurring sound sources are quickly parsed into sound object representations--which are then available for top-down selective attention. Here, we demonstrate that attention can be guided toward 1 of those representations, thereby attenuating change deafness. Furthermore, the effects of retro-cues in audition extend analogous findings in the visual domain, thereby suggesting that orienting attention to an object within visual or auditory STM may follow similar processing principles.

The use of relative weights to assess perceptual segregation in a concurrent profile analysis task

A series of experiments were conducted to address the need of a psychophysical tool to measure the perceptual segregation of concurrent sources. The experiments measured listeners' sensitivity to the spectral profile of a target sound embedded in a concurrent masker. Both the target and masker were harmonic complexes, which were presented at different fundamental frequencies in order to investigate the effects of this acoustic cue on segregation. The task was designed so that it either strongly encouraged the segregation of the two complexes (task-driven design) or it did not necessarily require segregation (listener-driven design). In both cases, the degree of segregation was assessed by deriving the relative decision weights on the target and masker. Larger differences between the target and masker weights were found as the fundamental frequency difference between the two complexes increased (0.5 – 15 semitones), suggesting that listeners were more successful in selectively attending to the target alone at larger fundamental frequency separations. Although quite different thresholds were obtained for the task-driven and listener-driven designs, the estimates of the decision weights were consistent across the two task designs, indicating that listeners' motivations did not influence the usefulness of the periodicity cue in segregating concurrent sounds.

Auditory–visual, positional, and semantic effects in visual extraction of slope

Extracting slopes from arrays of visual features is crucial for interpreting graphs. To understand broader influences on slope perception, we investigated the effects of concurrent sounds, of relative graph location, and of semantic priming on a visual search task in which observers searched for a graph with a positive or negative slope. Four bar graphs or scatter plots were simultaneously presented in separate quadrants of a visual display. Participants pressed a key as quickly as possible if one of the graphs displayed the target slope and otherwise refrained from response. A concurrently presented ascending pitch slowed responses to negative-slope targets, and concurrently presented descending pitch slowed responses to positive-slope targets, indicating crossmodal interference. This interference was eliminated when the sounds were presented 200 ms before the graphs, consistent with crossmodal interaction rather than response bias. Positive slopes were detected slowest in the upper-left quadrant whereas negative slopes were detected slowest in the upper-right quadrant, suggesting that slope detection is impeded when a graph is placed inconsistently with a mental number-line representation (negative values on the left and positive values on the right). Finally, positive slopes were detected faster when the search display was immediately preceded by a briefly flashed word ‘uphill’ compared to the word ‘downhill’ (and the converse for negative slopes), indicating a semantic priming effect, but this was observed only for scatter plots (the stimulus specificity precluding response bias). In summary, perception of visual slope is systematically influenced by auditory signals, by location of graphs, and by semantic priming.

The processing of concurrent sounds based on inharmonicity and asynchronous onsets: An object-related negativity (ORN) study

This study addresses the processing of concurrent sounds based on inharmonicity and onset asynchrony cues. We used harmonic complex sounds with one component starting marginally (40ms) or considerably (500ms) earlier than the complex and being slightly (3%) or strongly (13%) inharmonic. To index sound segregation of concurrent events, we measured the object-related negativity (ORN) component of the event-related potential. We contrasted two hypotheses: According to the concurrent-segregation hypothesis, increased onset asynchrony is assumed to promote segregation of the leading partial from the harmonic complex, which should be reflected in increased ORN amplitudes. That is, even with large onset asynchronies concurrent events would be processed by a simultaneous sound segregation mechanism. According to the sequential-integration hypothesis, however, with increased onset asynchrony concurrent cues are assumed to be less considered by simultaneous grouping processes, which should be reflected in attenuated ORN amplitudes for long onset asynchronies. This assumption is based on the notion that due to sequential integration, a stable percept of the leading partial has been developed within ~350ms after sound onset, thus less processing is required from scene analysis mechanisms based on concurrent cues. Indeed, with increased onset asynchrony ORN was found to decrease, which supports the sequential-integration hypothesis. In line with previous data, ORN was also found to increase with increased inharmonicity. The absence of an inharmonicity×onset asynchrony interaction further supports the assumption that both cues are used in different ways for simultaneous sound segregation.

Spatial Localization of Concurrent Multiple Sound Sources Using Phase Candidate Histogram

Mobile robots communicating with people would benefit from being able to detect sound sources to help localize interesting events in real-life settings. We propose using a spherical robot with four microphones to determine the spatial locations of multiple sound sources in ordinary rooms. The arrival temporal disparities from phase difference histograms are used to calculate the time differences. A precedence effect model suppresses the influence of echoes in reverberant environments. To integrate spatial cues of different microphones, we map the correlation between different microphone pairs on a 3D map corresponding to the azimuth and elevation of sound source direction. Results of experiments indicate that our proposed system provides sound source distribution very clearly and precisely, even concurrently in reverberant environments with the Echo Avoidance (EA) model.

Automatic Transcription of Polyphonic Music Based on the Constant-Q Bispectral Analysis

In the area of music information retrieval (MIR), automatic music transcription is considered one of the most challenging tasks, for which many different techniques have been proposed. This paper presents a new method for polyphonic music transcription: a system that aims at estimating pitch, onset times, durations, and intensity of concurrent sounds in audio recordings, played by one or more instruments. Pitch estimation is carried out by means of a front-end that jointly uses a constant-Q and a bispectral analysis of the input audio signal; subsequently, the processed signal is correlated with a fixed 2-D harmonic pattern. Onsets and durations detection procedures are based on the combination of the constant-Q bispectral analysis with information from the signal spectrogram. The detection process is agnostic and it does not need to take into account musicological and instrumental models or other a priori knowledge. The system has been validated against the standard Real-World Computing (RWC)-Classical Audio Database. The proposed method has demonstrated good performances in the multiple F0 tracking task, especially for piano-only automatic transcription at MIREX 2009.

Finding Your Mate at a Cocktail Party: Frequency Separation Promotes Auditory Stream Segregation of Concurrent Voices in Multi-Species Frog Choruses

Vocal communication in crowded social environments is a difficult problem for both humans and nonhuman animals. Yet many important social behaviors require listeners to detect, recognize, and discriminate among signals in a complex acoustic milieu comprising the overlapping signals of multiple individuals, often of multiple species. Humans exploit a relatively small number of acoustic cues to segregate overlapping voices (as well as other mixtures of concurrent sounds, like polyphonic music). By comparison, we know little about how nonhuman animals are adapted to solve similar communication problems. One important cue enabling source segregation in human speech communication is that of frequency separation between concurrent voices: differences in frequency promote perceptual segregation of overlapping voices into separate “auditory streams” that can be followed through time. In this study, we show that frequency separation (ΔF) also enables frogs to segregate concurrent vocalizations, such as those routinely encountered in mixed-species breeding choruses. We presented female gray treefrogs (Hyla chrysoscelis) with a pulsed target signal (simulating an attractive conspecific call) in the presence of a continuous stream of distractor pulses (simulating an overlapping, unattractive heterospecific call). When the ΔF between target and distractor was small (e.g., ≤3 semitones), females exhibited low levels of responsiveness, indicating a failure to recognize the target as an attractive signal when the distractor had a similar frequency. Subjects became increasingly more responsive to the target, as indicated by shorter latencies for phonotaxis, as the ΔF between target and distractor increased (e.g., ΔF = 6–12 semitones). These results support the conclusion that gray treefrogs, like humans, can exploit frequency separation as a perceptual cue to segregate concurrent voices in noisy social environments. The ability of these frogs to segregate concurrent voices based on frequency separation may involve ancient hearing mechanisms for source segregation shared with humans and other vertebrates.

Auditory enhancement of visual perception at threshold depends on visual abilities

Whether or not multisensory interactions can improve detection thresholds, and thus widen the range of perceptible events is a long-standing debate. Here we revisit this question, by testing the influence of auditory stimuli on visual detection threshold, in subjects exhibiting a wide range of visual-only performance. Above the perceptual threshold, crossmodal interactions have indeed been reported to depend on the subject's performance when the modalities are presented in isolation. We thus tested normal-seeing subjects and short-sighted subjects wearing their usual glasses. We used a paradigm limiting potential shortcomings of previous studies: we chose a criterion-free threshold measurement procedure and precluded exogenous cueing effects by systematically presenting a visual cue whenever a visual target (a faint Gabor patch) might occur. Using this carefully controlled procedure, we found that concurrent sounds only improved visual detection thresholds in the sub-group of subjects exhibiting the poorest performance in the visual-only conditions. In these subjects, for oblique orientations of the visual stimuli (but not for vertical or horizontal targets), the auditory improvement was still present when visual detection was already helped with flanking visual stimuli generating a collinear facilitation effect. These findings highlight that crossmodal interactions are most efficient to improve perceptual performance when an isolated modality is deficient.

Audiovisual contrast enhancement is articulated primarily via the M-pathway

Although it has been previously reported that audiovisual integration can modulate performance on some visual tasks, multisensory interactions have not been explicitly assessed in the context of different visual processing pathways. In the present study, we test auditory influences on visual processing employing a psychophysical paradigm that reveals distinct spatial contrast signatures of magnocellular and parvocellular visual pathways. We found that contrast thresholds are reduced when noninformative sounds are presented with transient, low-frequency Gabor patch stimuli and thus favor the M-system. In contrast, visual thresholds are unaffected by concurrent sounds when detection is primarily attributed to P-pathway processing. These results demonstrate that the visual detection enhancement resulting from multisensory integration is mainly articulated by the magnocellular system, which is most sensitive at low spatial frequencies. Such enhancement may subserve stimulus-driven processes including the orientation of spatial attention and fast, automatic ocular and motor responses. This dissociation helps explain discrepancies between the results of previous studies investigating visual enhancement by sounds.

Neural Correlates of Auditory Scene Analysis Based on Inharmonicity in Monkey Primary Auditory Cortex

Segregation of concurrent sounds in complex acoustic environments is a fundamental feature of auditory scene analysis. A powerful cue used by the auditory system to segregate concurrent sounds, such as speakers' voices at a cocktail party, is inharmonicity. This can be demonstrated when a component of a harmonic complex tone is perceived as a separate tone "popping out" from the complex as a whole when it is sufficiently mistuned from its harmonic value. The neural bases of perceptual "pop out" of mistuned harmonics are unclear. We recorded multiunit activity from primary auditory cortex (A1) of behaving monkeys elicited by harmonic complex tones that were either "in tune" or that contained a mistuned third harmonic set at the best frequency of the neural populations. Responses to mistuned sounds were enhanced relative to responses to "in-tune" sounds, thus correlating with the enhanced perceptual salience of the mistuned component. Consistent with human psychophysics of "pop out," response enhancements increased with the degree of mistuning, were maximal for neural populations tuned to the frequency of the mistuned component, and were not observed under comparable stimulus conditions that do not elicit perceptual "pop out." Mistuning was also associated with changes in neuronal temporal response patterns phase locked to "beats" in the stimuli. Intracortical auditory evoked potentials paralleled noninvasive neurophysiological correlates of perceptual "pop out" in humans, further augmenting the translational relevance of the results. Findings suggest two complementary neural mechanisms for "pop out," based on the detection of local differences in activation level or coherence of temporal response patterns across A1.

Human Auditory Cortex Activity Shows Additive Effects of Spectral and Spatial Cues during Speech Segregation

In noisy social gatherings, listeners perceptually integrate sounds originating from one person's voice (e.g., fundamental frequency (f(0)) and harmonics) at a particular location and segregate these from concurrent sounds of other talkers. Though increasing the spectral or the spatial distance between talkers promotes speech segregation, synergetic effects of spatial and spectral distances are less well understood. We studied how spectral and/or spatial distances between 2 simultaneously presented steady-state vowels contribute to perception and activation in auditory cortex using magnetoencephalography. Participants were more accurate in identifying both vowels when they differed in f(0) and location than when they differed in a single cue only or when they shared the same f(0) and location. The combined effect of f(0) and location differences closely matched the sum of single effects. The improvement in concurrent vowel identification coincided with an object-related negativity that peaked at about 140 ms after vowel onset. The combined effect of f(0) and location closely matched the sum of the single effects even though vowels with different f(0), location, or both generated different time courses of neuromagnetic activity. We propose that during auditory scene analysis, acoustic differences among the various sources are combined linearly to increase the perceptual distance between the co-occurring sound objects.

Responses in the Inferior Colliculus of the Guinea Pig to Concurrent Harmonic Series and the Effect of Inactivation of Descending Controls

One of the fundamental questions of auditory research is how sounds are segregated because, in natural environments, multiple sounds tend to occur at the same time. Concurrent sounds, such as two talkers, physically add together and arrive at the ear as a single input sound wave. The auditory system easily segregates this input into a coherent perception of each of the multiple sources. A common feature of speech and communication calls is their harmonic structure and in this report we used two harmonic complexes to study the role of the corticofugal pathway in the processing of concurrent sounds. We demonstrate that, in the inferior colliculus (IC) of the anesthetized guinea pig, deactivation of the auditory cortex altered the temporal and/or the spike response to the concurrent, monaural harmonic complexes. More specifically, deactivating the auditory cortex altered the representation of the relative level of the complexes. This suggests that the auditory cortex modulates the representation of the level of two harmonic complexes in the IC. Since sound level is a cue used in the segregation of auditory input, the corticofugal pathway may play a role in this segregation.

Visual determinants of a cross-modal illusion

Contrary to the predictions of established theory, Schutz and Lipscomb (2007) have shown that visual information can influence the perceived duration of concurrent sounds. In the present study, we deconstruct the visual component of their illusion, showing that (1) cross-modal influence depends on visible cues signaling an impact event (namely, a sudden change of direction concurrent with tone onset) and (2) the illusion is controlled primarily by the duration of post-impact motion. Other aspects of the post-impact motion--distance traveled, velocity, acceleration, and the rate of its change (i.e., its derivative, jerk)--play a minor role, if any. Together, these results demonstrate that visual event duration can influence the perception of auditory event duration, but only when stimulus cues are sufficient to give rise to the perception of a causal cross-modal relationship. This refined understanding of the illusion's visual aspects is helpful in comprehending why it contrasts so markedly with previous research on cross-modal integration, demonstrating that vision does not appreciably influence auditory judgments of event duration (Walker & Scott, 1981).