Simultaneous Speech Signals Research Articles

The role of unfamiliar accents in competing speech.

A listeners' ability to comprehend one speaker against a background of other speech-a phenomenon dubbed the cocktail party problem-varies according to the properties of the speech streams and the listener. Although a number of factors that contribute to a listener's ability to successfully segregate two simultaneous speech signals have been identified, comparably little work has focused on the role accents may play in this process. To this end, familiar Canadian-accented voices and unfamiliar British-accented voices were used in a competing talker task. Native speakers of Canadian English heard two different talkers simultaneously read sentences in the form of "[command] [colour] [preposition] [letter] [number] [adverb]" (e.g., "Lay blue at C4 now") and reported the coordinate from a target talker. Results indicate that on all but the most challenging trials, listeners did best when attending to an unfamiliar-accented target against a familiarly-accented masker and performed worse when forced to ignore the unfamiliar accent. These results suggest listeners can easily tune out a familiar accent, but are unable to do the same with an unfamiliar accent, indicating that unfamiliar accents are more effective maskers.

Significance of incorporating excitation source parameters for improved emotion recognition from speech and electroglottographic signals

The work presented in this paper explores the effectiveness of incorporating the excitation source parameters such as strength of excitation and instantaneous fundamental frequency ( $$F_0$$ ) for emotion recognition task from speech and electroglottographic (EGG) signals. The strength of excitation (SoE) is an important parameter indicating the pressure with which glottis closes at the glottal closure instants (GCIs). The SoE is computed by the popular zero frequency filtering (ZFF) method which accurately estimates the glottal signal characteristics by attenuating or removing the high frequency vocaltract interactions in speech. The arbitrary impulse sequence, obtained from the estimated GCIs, is used to derive the instantaneous $$F_0$$ . The SoE and the instantaneous $$F_0$$ parameters are combined with the conventional mel frequency cepstral coefficients (MFCC) to improve the recognition rates of distinct emotions (Anger, Happy and Sad) using Gaussian mixture models as classifier. The performances of the proposed combination of SoE and instantaneous $$F_0$$ and their dynamic features with MFCC coefficients are compared with the emotion utterances (4 emotions and neutral) from classical German full blown emotion speech database (EmoDb) having simultaneous speech and EGG signals and Surrey Audio Visual Expressed Emotion database (3 emotions and neutral) for both speaker dependent and speaker independent emotion recognition scenarios. To reinforce the effectiveness of the proposed features and for better statistical consistency of the emotion analysis, a fairly large emotion speech database of 220 utterances per emotion in Tamil language with simultaneous EGG recordings, is used in addition to EmoDb. The effectiveness of SoE and instantaneous $$F_0$$ in characterizing different emotions is also confirmed by the improved emotion recognition performance in Tamil speech-EGG emotion database.

Source-blind localization and segregation model featuring head movement and reflection removal

This model takes two simultaneous speech signals, spatialized to unique azimuth positions and convolved with a simple multi-tap stereo impulse response. The model first identifies reflections and generates an inversion filter for the left and right channels. It then localizes the sources and virtually rotates its head to a known orientation for the best resulting segregation of the sources. Next, the model segments the input signals in time and frequency, applies the inverse filter, and searches for residual energy in each bin to compensate the target signal from the mixture. From the residual non-canceled energy, it generates a binary masking map and overlays this on the mixed signal’s spectrogram to extract only the target signal. Improvement in SNR from head rotation approaches over 30 dB.

A binaural model using equalization/cancellation and simulated head movements to localize and extract one speaker from a mixture

This model takes a mixture of two simultaneous speech signals at unique azimuth positions to extract either speaker using the equalization/cancelation (EC) method. Head-related transfer functions are used to spatialize the two sound sources. The model localizes the sources by analyzing interaural time differences and then virtually rotates its head to find the position for the best signal-to-noise ratio. Next, the model segments the mixed speech signal in time and frequency bins, and uses an EC algorithm in each bin to compensate the target signal from the mixture. From the residual non-cancelled energy, it generates a binary map and overlays this on the spectrogram. The ability of the model to cancel out the target signal determines the bins where the target is actually present. The signal is then reconstructed in time and frequency, leaving only one desired target signal. The model achieves signal-to-noise ratios of up to 80 dB. [This material is based upon work supported by the National Science Foundation under Grant Nos. 1320059 and 1539276.]

Two-microphone spatial filtering provides speech reception benefits for cochlear implant users in difficult acoustic environments.

This article introduces and provides an assessment of a spatial-filtering algorithm based on two closely-spaced (∼1 cm) microphones in a behind-the-ear shell. The evaluated spatial-filtering algorithm used fast (∼10 ms) temporal-spectral analysis to determine the location of incoming sounds and to enhance sounds arriving from straight ahead of the listener. Speech reception thresholds (SRTs) were measured for eight cochlear implant (CI) users using consonant and vowel materials under three processing conditions: An omni-directional response, a dipole-directional response, and the spatial-filtering algorithm. The background noise condition used three simultaneous time-reversed speech signals as interferers located at 90°, 180°, and 270°. Results indicated that the spatial-filtering algorithm can provide speech reception benefits of 5.8 to 10.7 dB SRT compared to an omni-directional response in a reverberant room with multiple noise sources. Given the observed SRT benefits, coupled with an efficient design, the proposed algorithm is promising as a CI noise-reduction solution.

A Portable robot audition software system for multiple simultaneous speech signals

Since a robot is deployed in various kinds of environments, the robot audition system should work with minimum prior information on environments to localize, separate and recognize utterances by multiple simultaneous talkers. For example, it should not assume either the number of speakers, the location of speakers for sound source separation (SSS), or specially tuned acoustic model for automatic speech recognition (ASR). We developed a portable robot audition that uses eight microphones installed on the surface of robot's body such as Honda ASIMO, and SIG‐2 and Robovie‐R2 at Kyoto University with different microphone configurations. The system integrates SSS and ASR by using the Missing‐Feature Theory. For SSS, we use Geometric Source Separation (GSS) [Parra 2002] and multichannel post‐filter [Cohen 2002] to separate each utterance. Since separated speech signals are distorted due to interfering talkers and sound source separation, multichannel post‐filter enhanced speech signals. At this process, we create a missing feature mask that specifies which acoustic features are reliable in time‐frequency domain. Multiband Julius [Nishimura 2004], a missing‐feature‐theory based ASR, uses this mask to avoid the influence of unreliable features in recognizing such distorted speech signals. The system demonstrated a waitress that accepts meal orders placed by three actual human talkers and a voice "Rock‐Scissors‐Paper" game referee that decides who wins the game.

Improvement of recognition of simultaneous speech signals using AV integration and scattering theory for humanoid robots

This paper presents a method to improve recognition of three simultaneous speech signals by a humanoid robot equipped with a pair of microphones. In such situations, sound separation and automatic speech recognition (ASR) of the separated speech signal are difficult, because the signal-to-noise ratio is quite low (around −3 dB) and noise is not stable due to interfering voices. To improve recognition of three simultaneous speech signals, two key ideas are introduced. One is two-layered audio–visual integration of both name (ID) and location, that is, speech and face recognition, and speech and face localization. The other is acoustical modeling of the humanoid head by scattering theory. Sound sources are separated in real-time by an active direction-pass filter (ADPF), which extracts sounds from a specified direction by using the interaural phase/intensity difference estimated by scattering theory. Since features of separated sounds vary according to the sound direction, multiple direction- and speaker-dependent acoustic models are used. The system integrates ASR results by using the sound direction and speaker information provided by face recognition as well as confidence measure of ASR results to select the best one. The resulting system shows an improvement of about 10% on average against recognition of three simultaneous speech signals, where three speakers were located around the humanoid on a 1 m radius half circle, one of them being in front of him (angle 0°) and the other two being at symmetrical positions (± θ) varying by 10° steps from 0° to 90°.

Effects of fundamental frequency and vocal-tract length changes on attention to one of two simultaneous talkers.

Three experiments used the Coordinated Response Measure task to examine the roles that differences in F0 and differences in vocal-tract length have on the ability to attend to one of two simultaneous speech signals. The first experiment asked how increases in the natural F0 difference between two sentences (originally spoken by the same talker) affected listeners' ability to attend to one of the sentences. The second experiment used differences in vocal-tract length, and the third used both F0 and vocal-tract length differences. Differences in F0 greater than 2 semitones produced systematic improvements in performance. Differences in vocal-tract length produced systematic improvements in performance when the ratio of lengths was 1.08 or greater, particularly when the shorter vocal tract belonged to the target talker. Neither of these manipulations produced improvements in performance as great as those produced by a different-sex talker. Systematic changes in both F0 and vocal-tract length that simulated an incremental shift in gender produced substantially larger improvements in performance than did differences in F0 or vocal-tract length alone. In general, shifting one of two utterances spoken by a female voice towards a male voice produces a greater improvement in performance than shifting male towards female. The increase in performance varied with the intonation patterns of individual talkers, being smallest for those talkers who showed most variability in their intonation patterns between different utterances.

Contralateral masking effects in dichotic listening with two competing talkers in the target ear

Some of the most influential experiments in selective auditory attention have been based on a dichotic cocktail party task where listeners are asked to respond to a speech signal presented to one ear while ignoring a simultaneous competing speech signal presented to the other ear. These experiments have generally shown that the intelligibility of a monaural speech signal is unimpeded by the presence of an interfering speech signal at the opposite ear. However, recent results in our laboratory indicate that listeners cannot ignore a speech signal at the unattended ear when two simultaneous speech signals are presented to the target ear. In this study, the intelligibility of a target phrase in a two-talker stimulus presented to one ear was measured monaurally and with a speech or noise signal in the opposite ear. Performance in this task was unaffected when noise was added to the unattended ear, but degraded substantially when speech was added to the unattended ear. These results suggest that there are strong interactions between the monaural processes that listeners use to segregate two spatially colocated voices and the binaural processes they use to segregate voices originating from different apparent locations in space. [Work supported by AFOSR.]

Informational and energetic masking effects in the perception of two simultaneous talkers.

Although most recent multitalker research has emphasized the importance of binaural cues, monaural cues can play an equally important role in the perception of multiple simultaneous speech signals. In this experiment, the intelligibility of a target phrase masked by a single competing masker phrase was measured as a function of signal-to-noise ratio (SNR) with same-talker, same-sex, and different-sex target and masker voices. The results indicate that informational masking, rather than energetic masking, dominated performance in this experiment. The amount of masking was highly dependent on the similarity of the target and masker voices: performance was best when different-sex talkers were used and worst when the same talker was used for target and masker. Performance did not, however, improve monotonically with increasing SNR. Intelligibility generally plateaued at SNRs below 0 dB and, in some cases, intensity differences between the target and masking voices produced substantial improvements in performance with decreasing SNR. The results indicate that informational and energetic masking play substantially different roles in the perception of competing speech messages.

Spatial Audio Displays for Speech Communications: A Comparison of Free Field and Virtual Acoustic Environments

The ability of listeners to detect, identify, and monitor multiple simultaneous speech signals was measured in free field and virtual acoustic environments. Factorial combinations of four variables, including audio condition, spatial condition, the number of speech signals, and the sex of the talker were employed using a within-subjects design. Participants were required to detect the presentation of a critical speech signal among a background of non-signal speech events. Results indicated that spatial separation increased the percentage of correctly identified critical speech signals as the number of competing messages increased. These outcomes are discussed in the context of designing binaural speech displays to enhance speech communication in aviation environments.

A comparison of maskers on spatially separated competing messages

The intelligibility of simultaneous speech signals from pairs of talkers was improved with spatial separation of talkers using synthetic means. The simultaneous speech signals of the talkers, male versus male, female versus female, and male versus female, were separated by 0, 22, 44, 90, 134, and 180 deg. A simple left–right (180-deg) signal separation condition was also measured. Four listeners responded to both diotic and spatially separated presentations over headphones. The listeners were randomly prompted by a call sign to respond to one of the two talkers, providing an intelligibility baseline of 0 dB signal-to-noise ratio for each test phrase. Speech maskers used were either (1) pink noise or (2) a speech babble presented over headphones or, (3) pink noise presented by loudspeakers. In general, the female talkers were the least intelligible and the male versus female pair were the most intelligible. Spatial separations of 90 deg, or more, provided up to a 25% increase in intelligibility.