Nonnative audiovisual speech perception in noise: dissociable effects of the speaker and listener.

Zilong Xie,Joel Snyder,Bharath Chandrasekaran,Han-Gyol Yi

doi:10.1371/journal.pone.0114439

Abstract

Nonnative speech poses a challenge to speech perception, especially in challenging listening environments. Audiovisual (AV) cues are known to improve native speech perception in noise. The extent to which AV cues benefit nonnative speech perception in noise, however, is much less well-understood. Here, we examined native American English-speaking and native Korean-speaking listeners' perception of English sentences produced by a native American English speaker and a native Korean speaker across a range of signal-to-noise ratios (SNRs;−4 to −20 dB) in audio-only and audiovisual conditions. We employed psychometric function analyses to characterize the pattern of AV benefit across SNRs. For native English speech, the largest AV benefit occurred at intermediate SNR (i.e. −12 dB); but for nonnative English speech, the largest AV benefit occurred at a higher SNR (−4 dB). The psychometric function analyses demonstrated that the AV benefit patterns were different between native and nonnative English speech. The nativeness of the listener exerted negligible effects on the AV benefit across SNRs. However, the nonnative listeners' ability to gain AV benefit in native English speech was related to their proficiency in English. These findings suggest that the native language background of both the speaker and listener clearly modulate the optimal use of AV cues in speech recognition.

Highlights

In an increasingly global world, nonnative speech is a common occurrence
The probability of the correct keyword identification was significantly higher for speech produced by the native American English speaker than that by the native Korean speaker (p,.001), The probability of the correct keyword identification was significantly higher for native American English listeners than native Korean listeners (p5.003)
signal-to-noise ratios (SNRs) interacted significantly with the other three fixed effects. These results suggest that intelligibility benefit from SNR increment was less for AV condition, for speech from the native Korean speaker, and for native Korean listeners

Summary

Introduction

In an increasingly global world, nonnative speech is a common occurrence. In the United States, for example, more than 35 million adults are nonnative speakers of English [1]. Speech sounds produced by nonnative speakers can deviate significantly from native norms at segmental or suprasegmental levels [2, 3]. These deviations pose a challenge to speech perception, especially in challenging listening environments such as noisy conditions [4]. While previous studies have extensively examined the role of visual cues in speech perception, the majority of them have focused on native speakers and listeners [5,6,7,8,9,10,11,12,13,14,15,16]. We examine the extent to which these cues impact the perception of nonnative speech, relative to native speech in noise. We examine how nonnative listeners process AV cues from native and non-native speakers

Methods

Results

Discussion

Conclusion