Native Speech Perception Research Articles

The developmental pattern of native and non-native speech perception during the 1st year of life in Japanese infants

Language development during the 1st year of life is characterized by perceptual attunement: following language-general perception, a decline in the perception of non-native phonemes and a parallel increase in or maintenance of the perception of native phonemes. While this general pattern is well established, there are still many gaps in the literature. First, most evidence documenting these patterns comes from “Minority world countries” with only a limited number of studies from “Majority world countries”, limiting the range of languages and contrasts assessed. Second, few studies test both the developmental patterns of native and non-native speech perception in the same group of infants, making it hard to draw conclusions on simultaneous decline in non-native and increase in native speech perception. Such limitations are in part due to the effort that goes into testing developing speech sound perception, where usually only discrimination of one contrast per infant can be tested at a time. The present study thus set out to assess the feasibility of assessing a given infant on their discrimination of two speech sound contrasts during the same lab visit. It leveraged the presence of documented patterns of the improvement of native and the decline of non-native phoneme discrimination abilities in Japanese, therefore assessing native and non-native speech perception in Japanese infants from 6 to 12 months of age. Results demonstrated that 76 % of infants contributed discrimination data for both contrasts. We found a decline in non-native speech perception evident in discrimination of the non-native /ɹ/-/l/ consonant contrast at 9–11, but not at 11–13 months of age. Additionally, a parallel increase in native speech perception was demonstrated evident in an absence of native phonemic vowel length discrimination at 6–7 and 9–11 months and a discrimination of this contrast at 11–13 months of age. These results, based on a simultaneous assessment of native and non-native speech perception in Japanese-learning infants, demonstrate the feasibility of assessing the discrimination of two contrasts in one testing session and corroborate theoretical proposals on two hallmarks of perceptual attunement: a decrease in non-native and a facilitation in native speech perception during the first year of life.

Linguistic modulation of the neural encoding of phonemes.

Speech comprehension entails the neural mapping of the acoustic speech signal onto learned linguistic units. This acousto-linguistic transformation is bi-directional, whereby higher-level linguistic processes (e.g. semantics) modulate the acoustic analysis of individual linguistic units. Here, we investigated the cortical topography and linguistic modulation of the most fundamental linguistic unit, the phoneme. We presented natural speech and "phoneme quilts" (pseudo-randomly shuffled phonemes) in either a familiar (English) or unfamiliar (Korean) language to native English speakers while recording functional magnetic resonance imaging. This allowed us to dissociate the contribution of acoustic vs. linguistic processes toward phoneme analysis. We show that (i) the acoustic analysis of phonemes is modulated by linguistic analysis and (ii) that for this modulation, both of acoustic and phonetic information need to be incorporated. These results suggest that the linguistic modulation of cortical sensitivity to phoneme classes minimizes prediction error during natural speech perception, thereby aiding speech comprehension in challenging listening situations.

Rhythmic modulation of prediction errors: A top-down gating role for the beta-range in speech processing.

Natural speech perception requires processing the ongoing acoustic input while keeping in mind the preceding one and predicting the next. This complex computational problem could be handled by a dynamic multi-timescale hierarchical inferential process that coordinates the information flow up and down the language network hierarchy. Using a predictive coding computational model (Precoss-β) that identifies online individual syllables from continuous speech, we address the advantage of a rhythmic modulation of up and down information flows, and whether beta oscillations could be optimal for this. In the model, and consistent with experimental data, theta and low-gamma neural frequency scales ensure syllable-tracking and phoneme-level speech encoding, respectively, while the beta rhythm is associated with inferential processes. We show that a rhythmic alternation of bottom-up and top-down processing regimes improves syllable recognition, and that optimal efficacy is reached when the alternation of bottom-up and top-down regimes, via oscillating prediction error precisions, is in the beta range (around 20-30 Hz). These results not only demonstrate the advantage of a rhythmic alternation of up- and down-going information, but also that the low-beta range is optimal given sensory analysis at theta and low-gamma scales. While specific to speech processing, the notion of alternating bottom-up and top-down processes with frequency multiplexing might generalize to other cognitive architectures.

Non-native disadvantage in spoken word recognition is due to lexical knowledge and not type/level of noise

Adverse listening conditions typically trigger the use of linguistic knowledge, which helps the listener compensate for the impoverished acoustic signal. It is not clear, however, whether linguistic knowledge facilitates non-native speech perception in the same way as it does native speech perception. This study examined how native listeners and non-native learners use distributional characteristics of the Mandarin lexicon for word recognition in noise. Monosyllabic Mandarin words varying in syllable token frequency and syllable-tone co-occurrence probabilities were mixed with speech-shaped noise and two-talker babble at two signal-to-noise ratios. Participants were asked to listen to the speech and type the perceived syllable and tone word. The results showed that two-talker babble was more disruptive than speech-shaped noise, and a lower signal-to-noise ratio was more detrimental than a higher signal-to-noise ratio. However, neither noise type or noise level affected non-native word recognition disproportionately. Syllable token frequency facilitated word recognition for both listener groups, but only the native listeners showed robust use of syllable-tone co-occurrence probabilities for word recognition. For all listeners, lexical proficiency correlated positively with word recognition performance. These results suggest that lexical proficiency accounts for the use of distributional knowledge regardless of the type or level of noise.

Behavioral Pattern Analysis between Bilingual and Monolingual Listeners’ Natural Speech Perception on Foreign-Accented English Language Using Different Machine Learning Approaches

Speech perception in an adverse background/noisy environment is a complex and challenging human process, which is made even more complicated in foreign-accented language for bilingual and monolingual individuals. Listeners who have difficulties in hearing are affected most by such a situation. Despite considerable efforts, the increase in speech intelligibility in noise remains elusive. Considering this opportunity, this study investigates Bengali–English bilinguals and native American English monolinguals’ behavioral patterns on foreign-accented English language considering bubble noise, gaussian or white noise, and quiet sound level. Twelve regular hearing participants (Six Bengali–English bilinguals and Six Native American English monolinguals) joined in this study. Statistical computation shows that speech with different noise has a significant effect (p = 0.009) on listening for both bilingual and monolingual under different sound levels (e.g., 55 dB, 65 dB, and 75 dB). Here, six different machine learning approaches (Logistic Regression (LR), Linear Discriminant Analysis (LDA), K-nearest neighbors (KNN), Naïve Bayes (NB), Classification and regression trees (CART), and Support vector machine (SVM)) are tested and evaluated to differentiate between bilingual and monolingual individuals from their behavioral patterns in both noisy and quiet environments. Results show that most optimal performances were observed using LDA by successfully differentiating between bilingual and monolingual 60% of the time. A deep neural network-based model is proposed to improve this measure further and achieved an accuracy of nearly 100% in successfully differentiating between bilingual and monolingual individuals.

A stacked auto-encoder with scaled conjugate gradient algorithm for Malayalam ASR

Automatic speech recognition (ASR) is entitled to automate natural speech perception and the processing mechanism through analysis in the linguistic and acoustic features of the speech signal. ASR for children is highly challenging due to their developing physical aspects and rapidly changing articulation features. Therefore, ASR for children is still at its infant level. In this work, a stacked multilayer auto-encoder (AE) network is designed for ASR of the Malayalam vowel, articulated by children in the age group of five to ten. The proposed network structured with an unsupervised pre-training followed by supervised training. The pre-training coupled with two layers of sparse auto-encoders and scaled conjugate gradient (SCG) algorithm used for back-propagation. The auto-encoders are used to pre-train the network in an unsupervised (self- supervised) manner with 40,500 features that include Mel frequency cepstral coefficients (MFCC) and its derivatives, spectrogram formants and zero crossing rate (ZCR). In the softmax layer, the pre-trained network retrained in a supervised manner with bottleneck features. Fine-tuning has been applied in the trained network to enhance its performance. The unsupervised and supervised layers are stacked together to form a comprehensive network. The designed network has shown an average accuracy of 97% in training and 89.5% accuracy in the test data-set.

Brain-behavior relationships in incidental learning of non-native phonetic categories

Research has implicated the left inferior frontal gyrus (LIFG) in mapping acoustic-phonetic input to sound category representations, both in native speech perception and non-native phonetic category learning. At issue is whether this sensitivity reflects access to phonetic category information per se or to explicit category labels, the latter often being required by experimental procedures. The current study employed an incidental learning paradigm designed to increase sensitivity to a difficult non-native phonetic contrast without inducing explicit awareness of the categorical nature of the stimuli. Functional MRI scans revealed frontal sensitivity to phonetic category structure both before and after learning. Additionally, individuals who succeeded most on the learning task showed the largest increases in frontal recruitment after learning. Overall, results suggest that processing novel phonetic category information entails a reliance on frontal brain regions, even in the absence of explicit category labels.

Does infant speech perception predict later vocabulary development in bilingual infants?

One of the most significant transitions reported in infant psychological development is perceptual narrowing whereby infants orient towards their native language. In monolingual infants, the progress made by infants in perceptual narrowing positively predicts later vocabulary size. The relationship between infant speech perception and later language development has not been thus far reported in bilingual populations. The present study investigated infant speech perception in relation to later vocabulary development in a prospective longitudinal study of bilingual infants over the first three years of life. Our study revealed three primary findings. First, unlike monolingual infants, bilingual infants demonstrated a positive correlation between native and non-native phonetic discrimination. Second, native speech perception at 10–11 months predicted single-language (English) vocabulary at 2 years, but not second language (Mandarin), nor total conceptual vocabulary. Third, infants with a native language orientation relative to a non-native orientation in phonetic discrimination at 10–11 months tended towards higher English vocabulary scores at 3 years of age. When placed in the context of prior research with monolingual infants, results with bilingual infants point to distinct relationships between native and non-native speech perception in infancy, but to a similar continuity between infant speech discrimination and later vocabulary growth.

Low-frequency cortical responses to natural speech reflect probabilistic phonotactics

Humans comprehend speech despite the various challenges such as mispronunciation and noisy environments. Our auditory system is robust to these thanks to the integration of the sensory input with prior knowledge and expectations built on language-specific regularities. One such regularity regards the permissible phoneme sequences, which determine the likelihood that a word belongs to a given language (phonotactic probability; “blick” is more likely to be an English word than “bnick”). Previous research demonstrated that violations of these rules modulate brain-evoked responses. However, several fundamental questions remain unresolved, especially regarding the neural encoding and integration strategy of phonotactics in naturalistic conditions, when there are no (or few) violations. Here, we used linear modelling to assess the influence of phonotactic probabilities on the brain responses to narrative speech measured with non-invasive EEG. We found that the relationship between continuous speech and EEG responses is best described when the stimulus descriptor includes phonotactic probabilities. This indicates that low-frequency cortical signals (<9 Hz) reflect the integration of phonotactic information during natural speech perception, providing us with a measure of phonotactic processing at the individual subject-level. Furthermore, phonotactics-related signals showed the strongest speech-EEG interactions at latencies of 100–500 ms, supporting a pre-lexical role of phonotactic information.

Relationship between Working Memory and Identification of a Few Native Phonetic Contrasts

Objectives: Large individual variability is documented for identification performance of native phones, especially in challenging situations. It is not known whether the ability to utilize cues available for phone identification is facilitated by cognitive abilities, thereby explaining a proportion of the individual variability. This study investigated the relationship between working memory capacity and identification of a few Malayalam phones in the absence of contextual cues among native listeners. Methods: Forty native listeners of Malayalam, aged between 18 and 25, participated in this study. Participants identified 8 Malayalam phones embedded in nonsense words. Working memory capacity was measured using tasks such as reading span, operation span, digit forward span, and digit backward span. Identification score for each phone, total phone identification score (average identification score from 8 phones), and reaction time during identification were obtained. Results: Phone identification score of participants ranged from 57.8% to 99%. Pearson product moment correlation analysis showed a significant positive correlation between all measures of working memory capacity and total phone identification score, indicating that working memory capacity play a role in the identification of phones. Reaction time showed a significant negative correlation with digit backward span and operation span. The measures of working memory capacity accounted for 24.7% of the variability in phone identification score. Conclusion: Identification of phones in the absence of contextual cues increases the cognitive load. Therefore, higher working memory capacity might aid in native phone identification in difficult situations. This study reveals the top down influence of cognition on native speech perception.

Cortical Tracking of Global and Local Variations of Speech Rhythm during Connected Natural Speech Perception.

During natural speech perception, listeners must track the global speaking rate, that is, the overall rate of incoming linguistic information, as well as transient, local speaking rate variations occurring within the global speaking rate. Here, we address the hypothesis that this tracking mechanism is achieved through coupling of cortical signals to the amplitude envelope of the perceived acoustic speech signals. Cortical signals were recorded with magnetoencephalography (MEG) while participants perceived spontaneously produced speech stimuli at three global speaking rates (slow, normal/habitual, and fast). Inherently to spontaneously produced speech, these stimuli also featured local variations in speaking rate. The coupling between cortical and acoustic speech signals was evaluated using audio-MEG coherence. Modulations in audio-MEG coherence spatially differentiated between tracking of global speaking rate, highlighting the temporal cortex bilaterally and the right parietal cortex, and sensitivity to local speaking rate variations, emphasizing the left parietal cortex. Cortical tuning to the temporal structure of natural connected speech thus seems to require the joint contribution of both auditory and parietal regions. These findings suggest that cortical tuning to speech rhythm operates on two functionally distinct levels: one encoding the global rhythmic structure of speech and the other associated with online, rapidly evolving temporal predictions. Thus, it may be proposed that speech perception is shaped by evolutionary tuning, a preference for certain speaking rates, and predictive tuning, associated with cortical tracking of the constantly changing-rate of linguistic information in a speech stream.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex.

During natural speech perception, humans must parse temporally continuous auditory and visual speech signals into sequences of words. However, most studies of speech perception present only single words or syllables. We used electrocorticography (subdural electrodes implanted on the brains of epileptic patients) to investigate the neural mechanisms for processing continuous audiovisual speech signals consisting of individual sentences. Using partial correlation analysis, we found that posterior superior temporal gyrus (pSTG) and medial occipital cortex tracked both the auditory and the visual speech envelopes. These same regions, as well as inferior temporal cortex, responded more strongly to a dynamic video of a talking face compared to auditory speech paired with a static face. Occipital cortex and pSTG carry temporal information about both auditory and visual speech dynamics. Visual speech tracking in pSTG may be a mechanism for enhancing perception of degraded auditory speech.

A Spatial Map of Onset and Sustained Responses to Speech in the Human Superior Temporal Gyrus

To derive meaning from speech, we must extract multiple dimensions of concurrent information from incoming speech signals. That is, equally important to processing phonetic features is the detection of acoustic cues that give structure and context to the information we hear. How the brain organizes this information is unknown. Using data-driven computational methods on high-density intracranial recordings from 27 human participants, we reveal the functional distinction of neural responses to speech in the posterior superior temporal gyrus according to either onset or sustained response profiles. Though similar response types have been observed throughout the auditory system, we found novel evidence for a major spatial parcellation in which a distinct caudal zone detects acoustic onsets and a rostral-surround zone shows sustained, relatively delayed responses to ongoing speech stimuli. While posterior onset and anterior sustained responses are used substantially during natural speech perception, they are not limited to speech stimuli and are seen even for reversed or spectrally rotated speech. Single-electrode encoding of phonetic features in each zone depended upon whether the sound occurred at sentence onset, suggesting joint encoding of phonetic features and their temporal context. Onset responses in the caudal zone could accurately decode sentence and phrase onset boundaries, providing a potentially important internal mechanism for detecting temporal landmarks in speech and other natural sounds. These findings suggest that onset and sustained responses not only define the basic spatial organization of high-order auditory cortex but also have direct implications for how speech information is parsed in the cortex. VIDEO ABSTRACT.

Atypical cortical entrainment to speech in the right hemisphere underpins phonemic deficits in dyslexia

Developmental dyslexia is a multifaceted disorder of learning primarily manifested by difficulties in reading, spelling, and phonological processing. Neural studies suggest that phonological difficulties may reflect impairments in fundamental cortical oscillatory mechanisms. Here we examine cortical mechanisms in children (6–12 years of age) with or without dyslexia (utilising both age- and reading-level-matched controls) using electroencephalography (EEG). EEG data were recorded as participants listened to an audio-story. Novel electrophysiological measures of phonemic processing were derived by quantifying how well the EEG responses tracked phonetic features of speech. Our results provide, for the first time, evidence for impaired low-frequency cortical tracking to phonetic features during natural speech perception in dyslexia. Atypical phonological tracking was focused on the right hemisphere, and correlated with traditional psychometric measures of phonological skills used in diagnostic dyslexia assessments. Accordingly, the novel indices developed here may provide objective metrics to investigate language development and language impairment across languages.

Audio-visual speech perception in adult readers with dyslexia: an fMRI study.

Developmental dyslexia is a specific deficit in reading and spelling that often persists into adulthood. In the present study, we used slow event-related fMRI and independent component analysis to identify brain networks involved in perception of audio-visual speech in a group of adult readers with dyslexia (RD) and a group of fluent readers (FR). Participants saw a video of a female speaker saying a disyllabic word. In the congruent condition, audio and video input were identical whereas in the incongruent condition, the two inputs differed. Participants had to respond to occasionally occurring animal names. The independent components analysis (ICA) identified several components that were differently modulated in FR and RD. Two of these components including fusiform gyrus and occipital gyrus showed less activation in RD compared to FR possibly indicating a deficit to extract face information that is needed to integrate auditory and visual information in natural speech perception. A further component centered on the superior temporal sulcus (STS) also exhibited less activation in RD compared to FR. This finding is corroborated in the univariate analysis that shows less activation in STS for RD compared to FR. These findings suggest a general impairment in recruitment of audiovisual processing areas in dyslexia during the perception of natural speech.

Searching High and Low: Prosodic Breaks Disambiguate Relative Clauses.

During natural speech perception, listeners rely on a wide range of cues to support comprehension, from semantic context to prosodic information. There is a general consensus that prosody plays a role in syntactic parsing, but most studies focusing on ambiguous relative clauses (RC) show that prosodic cues, alone, are insufficient to reverse the preferred interpretation of sentence. These findings suggest that universally preferred structures (e.g., Late Closure principle) matter far more than prosodic cues in such cases. This study explores an alternative hypothesis: that the weak effect of prosody might be due to the influence of various syntactic, lexical-semantic, and acoustic confounding factors, and investigate the consequences of prosodic breaks while controlling these variables. We used Spanish RC sentences in three experimental conditions where the presence and position (following the first or second noun phrase) of prosodic breaks was manipulated. The results showed that the placement of a prosodic break determined sentence interpretation by changing the preferred attachment of the RC. Listeners’ natural preference for low attachment (in the absence of break) was reinforced when a prosodic break was placed after the first noun. In contrast, a prosodic break placed after the second noun reversed the preferred interpretation of the sentence, toward high attachment. We argue that, in addition to other factors, listeners indeed use prosodic breaks as robust cues to syntactic parsing during speech processing, as these cues may direct listeners toward one interpretation or another.

Neurobiological constraints on speech learning: Individual differences and plasticity

A primary goal of auditory neuroscience is to understand how conspecific sounds are represented, learned, and mapped to constructs. Speech signals, conspecific to humans, are multidimensional, acoustically variable, and temporally ephemeral. A significant computational challenge in speech perception (and more broadly, audition) is categorization, that is, mapping continuous, multidimensional, and variable acoustic signals into discrete, behavioral equivalence classes. Despite the enormity of this computational challenge, native speech perception is rapid and automatic. In contrast, learning novel speech categories is effortful, and considered one of the most challenging categorization tasks for the mature brain. I will discuss three lines of ongoing research using multimodal neuroimaging, computational modeling, and behavioral training approaches: a) examine the multiple neural systems underlying successful L2 speech category learning in adulthood, b) assess sources of individual differences in L2 speech category learning, and c) design optimal, neurobiologically constrained training paradigms that reduce inter-individual differences in L2 speech category learning. These studies will provide insights into these fundamental questions: consistent with prior work on visual category learning, are multiple neural systems involved in speech categorization? What is the role of emerging expertise and individual differences in mediating neural and computational processes involved in speech category learning?

Subjective ratings of masker disturbance during the perception of native and non-native speech

The aim of the present study was to address how 43 normal-hearing (NH) and hearing-impaired (HI) listeners subjectively experienced the disturbance generated by four masker conditions (i.e., stationary noise, fluctuating noise, Swedish two-talker babble and English two-talker babble) while listening to speech in two target languages, i.e., Swedish (native) or English (non-native). The participants were asked to evaluate their noise-disturbance experience on a continuous scale from 0 to 10 immediately after having performed each listening condition. The data demonstrated a three-way interaction effect between target language, masker condition, and group (HI versus NH). The HI listeners experienced the Swedish-babble masker as significantly more disturbing for the native target language (Swedish) than for the non-native language (English). Additionally, this masker was significantly more disturbing than each of the other masker types during the perception of Swedish target speech. The NH listeners, on the other hand, indicated that the Swedish speech-masker was more disturbing than the stationary and the fluctuating noise-maskers for the perception of English target speech. The NH listeners perceived more disturbance from the speech maskers than the noise maskers. The HI listeners did not perceive the speech maskers as generally more disturbing than the noise maskers. However, they had particular difficulty with the perception of native speech masked by native babble, a common condition in daily-life listening conditions. These results suggest that the characteristics of the different maskers applied in the current study seem to affect the perceived disturbance differently in HI and NH listeners. There was no general difference in the perceived disturbance across conditions between the HI listeners and the NH listeners.

Discrimination of Multiple Coronal Stop Contrasts in Wubuy (Australia): A Natural Referent Consonant Account

Native speech perception is generally assumed to be highly efficient and accurate. Very little research has, however, directly examined the limitations of native perception, especially for contrasts that are only minimally differentiated acoustically and articulatorily. Here, we demonstrate that native speech perception may indeed be more difficult than is often assumed, where phonemes are highly similar, and we address the nature and extremes of consonant perception. We present two studies of native and non-native (English) perception of the acoustically and articulatorily similar four-way coronal stop contrast /t ʈ ȶ/ (apico-alveolar, apico-retroflex, lamino-dental, lamino-alveopalatal) of Wubuy, an indigenous language of Australia. The results show that all listeners find contrasts involving /ȶ/ easy to discriminate, but that, for both groups, contrasts involving /t ʈ / are much harder. Where the two groups differ, the results largely reflect native language (Wubuy vs English) attunement as predicted by the Perceptual Assimilation Model [1, 2, 3]. We also observe striking perceptual asymmetries in the native listeners' perception of contrasts involving the latter three stops, likely due to the differences in input frequency. Such asymmetries have not previously been observed in adults, and we propose a novel Natural Referent Consonant Hypothesis to account for the results.

Mandarin Chinese vowel-plus-tone identification in noise: Effects of language experience

Several studies found better English vowel identification in English multi-talker babble (MTB) and temporally-modulated (TM) noise, but not in quiet condition for native Chinese listeners in the US (CNU) with the US residency of 1–3 years than native Chinese listeners in China (CNC) with no residency history in English speaking countries. Two possible explanations were proposed: (1) CNU listeners used temporal dips of noise more efficiently than CNC listeners; and (2) CNU listeners had less informational masking of MTB than their CNC peers. The current study explored whether the difference in noise processing between CNU and CNC listeners was also presented for their native speech perception. Chinese vowel-plus-tone identifications were measured for CNU and CNC in quiet, stationary and TM noise, babble-modulated noise, and MTB. The identification scores of CNU listeners were significantly higher than CNC listeners in most noisy backgrounds, whereas both groups had the same performance in quiet. Moreover, compared with CNC listeners, CNU listeners gained greater masking releases from the temporal modulation in TM noise at low SNRs, whereas no significant difference was found in informational masking between the two groups. In conclusion, the native English experience may improve native Chinese listeners' capacity to use temporal glimpses in TM noise, possibly depending on the modulation frequency and depth, while it may not improve their ability to resist the informational masking of babble when perceiving their native speech.