Human Speech Perception Research Articles

Release from same-talker speech-in-speech masking: Effects of masker intelligibility and other contributing factorsa).

Human speech perception declines in the presence of masking speech, particularly when the masker is intelligible and acoustically similar to the target. A prior investigation demonstrated a substantial reduction in masking when the intelligibility of competing speech was reduced by corrupting voiced segments with noise [Huo, Sun, Fogerty, and Tang (2023), "Quantifying informational masking due to masker intelligibility in same-talker speech-in-speech perception," in Interspeech 2023, pp. 1783-1787]. As this processing also reduced the prominence of voiced segments, it was unclear whether the unmasking was due to reduced linguistic content, acoustic similarity, or both. The current study compared the masking of original competing speech (high intelligibility) to competing speech with time reversal of voiced segments (VS-reversed, low intelligibility) at various target-to-masker ratios. Modeling results demonstrated similar energetic masking between the two maskers. However, intelligibility of the target speech was considerably better with the VS-reversed masker compared to the original masker, likely due to the reduced linguistic content. Further corrupting the masker's voiced segments resulted in additional release from masking. Acoustic analyses showed that the portion of target voiced segments overlapping with masker voiced segments and the similarity between target and masker overlapped voiced segments impacted listeners' speech recognition. Evidence also suggested modulation masking in the spectro-temporal domain interferes with listeners' ability to glimpse the target.

Two-stage algorithm of spectral analysis for automatic speech recognition systems

Within the framework of a dynamically developing direction of research in the field of acoustic measurements, the task of spectral analysis of speech signals in automatic speech recognition systems is considered. The low efficiency of the systems in unfavorable speech production conditions (noise, insufficient intelligibility of speech sounds) compared to human perception of oral speech is noted. To improve the efficiency of automatic speech recognition systems, a two-stage algorithm for spectral analysis of speech signals is proposed. The first stage of speech signal processing consists of its parametric spectral analysis using an autoregressive model of the vocal tract of a conditional speaker. The second stage of processing is the transformation (modification) of the obtained spectral estimate according to the principle of frequency-selective amplification of the amplitude of the main formants of the intra-periodic power spectrum. The software implementation of the proposed algorithm based on the high-speed computational procedure of the fast Fourier transform is described. Using the author’s software, a full-scale experiment was carried out: an additive mixture of vowel sounds of the control speaker’s speech with white Gaussian noise was studied. Based on the results of the experiment, it was concluded that the amplitude of the main speech signal formants were amplified by 10–20 dB and, accordingly, a significant improvement in the speech sounds intelligibility. The scope of possible application of the developed algorithm covers automatic speech recognition systems based on speech signal processing in the frequency domain, including the use of artificial neural networks.

Time scale of adaptation at the tonal sequence processing in the awake mice auditory cortex neurons

The study was firstly carried out on stimulus-specific adaptation of neurons in the primary and anterior fields of the awake house mice auditory cortex to sound sequences of four 100-ms tonal signals, with frequency of tones corresponding to the neuronal characteristic frequency, and also with the inter-tone interval constant for one sequence and varying from 0 to 2000 ms in different sequences. The analysis of the data obtained showed the adaptation effect in the responses of all studied primary auditory cortex neurons, which was observed as the absence or significant decrease in activity evoked by the components of a series of tones following the 1st, at inter-stimulus intervals of 0–500 ms. A quantitative assessment of the adaptation effects as a function of inter-stimulus intervals within the tonal sequence, performed over whole population of studied neurons, showed that the individual time scales of adaptation of neurons varied significantly, which may be crucial for the formation of optimal time windows for the processing of grouping and separation of sound events, which are important both for perception of animal vocalizations and human speech.

Developmental hearing loss–induced perceptual deficits are rescued by genetic restoration of cortical inhibition

Even a transient period of hearing loss during the developmental critical period can induce long-lasting deficits in temporal and spectral perception. These perceptual deficits correlate with speech perception in humans. In gerbils, these hearing loss-induced perceptual deficits are correlated with a reduction of both ionotropic GABAA and metabotropic GABAB receptor-mediated synaptic inhibition in auditory cortex, but most research on critical period plasticity has focused on GABAA receptors. Therefore, we developed viral vectors to express proteins that would upregulate gerbil postsynaptic inhibitory receptor subunits (GABAA, Gabra1; GABAB, Gabbr1b) in pyramidal neurons, and an enzyme that mediates GABA synthesis (GAD65) presynaptically in parvalbumin-expressing interneurons. A transient period of developmental hearing loss during the auditory critical period significantly impaired perceptual performance on two auditory tasks: amplitude modulation depth detection and spectral modulation depth detection. We then tested the capacity of each vector to restore perceptual performance on these auditory tasks. While both GABA receptor vectors increased the amplitude of cortical inhibitory postsynaptic potentials, only viral expression of postsynaptic GABAB receptors improved perceptual thresholds to control levels. Similarly, presynaptic GAD65 expression improved perceptual performance on spectral modulation detection. These findings suggest that recovering performance on auditory perceptual tasks depends on GABAB receptor-dependent transmission at the auditory cortex parvalbumin to pyramidal synapse and point to potential therapeutic targets for developmental sensory disorders.

A module for automatic analysis of burst spectra for consonant place detection

Widespread methods for automatic speech processing have become increasingly powerful, but they do not aim to model aspects of human speech processing. Previous research has shown that individual acoustic cues are important in human speech perception. Developing a model that is capable of identifying individual acoustic cues in speech enhances our ability to extract meaningful information from a speech signal, irrespective of speaker variations or phonemic differences, in a way that provides a transparent and testable model of human speech processing. This research investigates a module for automatic analysis of the spectral burst cue in fricative and plosive speech sounds. The method determines the place of articulation of the spectral burst by utilizing spectral moment measurements near locations where a spectral burst is likely to occur as features in a Gaussian mixture model. This research lays the groundwork for a dynamic model that is developed to be consistent with the Bayesian belief updating framework, in alignment with prior work in human speech perception.

Visual Facial Enhancements Can Significantly Improve Speech Perception in the Presence of Noise.

Human speech perception is generally optimal in quiet environments, however it becomes more difficult and error prone in the presence of noise, such as other humans speaking nearby or ambient noise. In such situations, human speech perception is improved by speech reading, i.e., watching the movements of a speaker's mouth and face, either consciously as done by people with hearing loss or subconsciously by other humans. While previous work focused largely on speech perception of two-dimensional videos of faces, there is a gap in the research field focusing on facial features as seen in head-mounted displays, including the impacts of display resolution, and the effectiveness of visually enhancing a virtual human face on speech perception in the presence of noise. In this paper, we present a comparative user study ( N=21) in which we investigated an audio-only condition compared to two levels of head-mounted display resolution (1832×1920 or 916×960 pixels per eye) and two levels of the native or visually enhanced appearance of a virtual human, the latter consisting of an up-scaled facial representation and simulated lipstick (lip coloring) added to increase contrast. To understand effects on speech perception in noise, we measured participants' speech reception thresholds (SRTs) for each audio-visual stimulus condition. These thresholds indicate the decibel levels of the speech signal that are necessary for a listener to receive the speech correctly 50% of the time. First, we show that the display resolution significantly affected participants' ability to perceive the speech signal in noise, which has practical implications for the field, especially in social virtual environments. Second, we show that our visual enhancement method was able to compensate for limited display resolution and was generally preferred by participants. Specifically, our participants indicated that they benefited from the head scaling more than the added facial contrast from the simulated lipstick. We discuss relationships, implications, and guidelines for applications that aim to leverage such enhancements.

Comparison of the prediction accuracy of machine learning algorithms in crosslinguistic vowel classification

Machine learning algorithms can be used for the prediction of nonnative sound classification based on crosslinguistic acoustic similarity. To date, very few linguistic studies have compared the classification accuracy of different algorithms. This study aims to assess how well machines align with human speech perception by assessing the ability of three machine learning algorithms, namely, linear discriminant analysis (LDA), decision tree (C5.0), and neural network (NNET), to predict the classification of second language (L2) sounds in terms of first language (L1) categories. The models were trained using the first three formants and duration of L1 vowels and fed with the same acoustic features of L2 vowels. To validate their accuracy, adult L2 speakers completed a perceptual classification task. The results indicated that NNET predicted with success the classification of all L2 vowels with the highest proportion in terms of L1 categories, while LDA and C5.0 missed only one vowel each. Furthermore, NNET exhibited superior accuracy in predicting the full range of above chance responses, followed closely by LDA. C5.0 did not meet the anticipated performance levels. The findings can hold significant implications for advancing both the theoretical and practical frameworks of speech acquisition.

Design of P-FLANN Model for Intelligent Water Fountain Sound Pleasantness Monitoring Using Bio-inspired Computing and Human Speech Perception

Cognitive-inspired Computational Computing systems play a crucial role in designing intelligent health monitoring systems which help both patients and hospitals. It also helps in early and consistent decision-making for various health issues including human psychological health. Water fountains built in parks and public spaces are used as decorative instruments which not only give appealing visuals but also it provides a relaxing environment to the visitors. These natural sounds have a direct effect on the psychological health of visitors. Very few research works are reported on developing the relationship between water sounds and their corresponding psychological impact. This assessment needs trained manpower and a lot of experimental time which is costly and may not be always available. In this paper to access the human health-friendly water fountain sounds from the pleasantness, a Perceptually Weighted functional link artificial neural network (P-FLANN) model is developed. To reduce the computational complexity of training and for faster convergence, swam intelligence-based optimization algorithm is used for updating the weights. It is observed from the comparative simulation results that the proposed P-FLANN model can effectively perform prediction tasks which is not only cost-effective but also 95% accurate and can play a crucial role in designing human health-friendly water fountains in smart cities.

Perceptual specializations for processing species-specific vocalizations in the common marmoset (Callithrix jacchus)

How humans and animals segregate sensory information into discrete, behaviorally meaningful categories is one of the hallmark questions in neuroscience. Much of the research around this topic in the auditory system has centered around human speech perception, in which categorical processes result in an enhanced sensitivity for acoustically meaningful differences and a reduced sensitivity for nonmeaningful distinctions. Much less is known about whether nonhuman primates process their species-specific vocalizations in a similar manner. We address this question in the common marmoset, a small arboreal New World primate with a rich vocal repertoire produced across a range of behavioral contexts. We first show that marmosets perceptually categorize their vocalizations in ways that correspond to previously defined call types for this species. Next, we show that marmosets are differentially sensitive to changes in particular acoustic features of their most common call types and that these sensitivity differences are matched to the population statistics of their vocalizations in ways that likely maximize category formation. Finally, we show that marmosets are less sensitive to changes in these acoustic features when within the natural range of variability of their calls, which possibly reflects perceptual specializations which maintain existing call categories. These findings suggest specializations for categorical vocal perception in a New World primate species and pave the way for future studies examining their underlying neural mechanisms.

Hearing Dogs and Seeing Barks: Multimodal Sensory Perception of Dogs

The field of multisensory integration in human speech perception has been well explored throughout a multitude of studies. Perceptual phenomena and models have been reported, including most notably, the McGurk effect and the Morton-Massaro law of information integration. Existing research of multimodal integration in human perception, include studies that involve perception of drinks and food. Beyond this, scientists have also studied how non-human species, such as dolphins and bats, integrate different sources of information. The purpose of the current study was to test whether multimodal sensory integration is used by humans when perceiving the size of dogs. Three separate audio-visual sources of barking dogs – sized small, medium, and large – were used. A 3x3 expanded factorial design was applied to test all possible pairings of auditory and visual combinations from the three sources. The stimuli were included on a questionnaire and then digitally distributed to participants, the majority of whom were high-school students aged 13-17. After viewing each stimulus, participants were prompted to answer what sized dog barked. The results of the current study were consistent with the McGurk effect. When the auditory-visual stimuli were consistent, participants were able to accurately identify what kind of dog barked. However, when the auditory-visual stimuli were inconsistent, the intermediate responses were most frequent. Thus, the results of this study are not consistent with the motor theory of perception. This study yields practical applications in the field of interspecies perception and human-canine communication.

Development of an algorithm for characterizing speech production patterns as context-based Cue Production Profiles

A given word or sound can take on a wide range of acoustic forms in conversational speech, and these patterns of variation are systematic with context. This substantial variability poses a problem for modeling human speech perception, but it can also be highly informative if it can be measured quantitatively; surface phonetic patterns can be used to characterize different speakers, dialects, speaking styles and speech disorders, as well as context-driven variation within a speaker. Adopting the view that individual acoustic cues provide a useful vocabulary for describing these variation patterns, we are developing an algorithm for tabulating the range of acoustic cue production patterns for each phoneme, in a corpus of recorded speech, with the potential for extension to words and phrases. This algorithm will enable the determination of key factors that govern systematic speech variation, including segmental, prosodic and lexical contexts, which can be used to create a profile for a speaker or defined population of speakers, i.e., a Cue Production Profile.

Can You Hear Me Now? Sensitive Comparisons of Human and Machine Perception.

The rise of machine-learning systems that process sensory input has brought with it a rise in comparisons between human and machine perception. But such comparisons face a challenge: Whereas machine perception of some stimulus can often be probed through direct and explicit measures, much of human perceptual knowledge is latent, incomplete, or unavailable for explicit report. Here, we explore how this asymmetry can cause such comparisons to misestimate the overlap in human and machine perception. As a case study, we consider human perception of adversarial speech - synthetic audio commands that are recognized as valid messages by automated speech-recognition systems but that human listeners reportedly hear as meaningless noise. In five experiments, we adapt task designs from the human psychophysics literature to show that even when subjects cannot freely transcribe such speech commands (the previous benchmark for human understanding), they can sometimes demonstrate other forms of understanding, including discriminating adversarial speech from closely matched nonspeech (Experiments 1 and 2), finishing common phrases begun in adversarial speech (Experiments 3 and 4), and solving simple math problems posed in adversarial speech (Experiment 5) - even for stimuli previously described as unintelligible to human listeners. We recommend the adoption of such "sensitive tests" when comparing human and machine perception, and we discuss the broader consequences of such approaches for assessing the overlap between systems.

Auditory pattern discrimination in budgerigars (Melopsittacus undulatus)

Auditory patterns carry information in human speech at multiple levels, including the surface relationships between sounds within words in phonology and the abstract structures of syntax. The sequences of other animal vocalizations, such as birdsong, can also be described as auditory patterns, but few studies have probed how the sequences are perceived at multiple levels. Past work shows that a small parrot species, the budgerigar (Melopsittacus undulatus), exceeds other birds in sequence perception and is even sensitive to abstract structure. But it is not known what level of auditory analysis is dominant in perception or what limits might exist in sensitivity to abstract structure. Here, budgerigars were tested on their ability to discriminate changes in an auditory pattern, AAB, i.e. sound-same different, to ask how they attended to surface relationships among the sounds and the abstract relationships of same/different among the elements. The results show that the budgerigars primarily used surface transitions between the sounds when discriminating the sequences, but were able to use the abstract relationships to a limited extent, largely restricted to two elements. This study provides insight into how budgerigars extract information from conspecific vocalizations and how their capacities compare to human speech perception.

Confusion2Vec 2.0: Enriching ambiguous spoken language representations with subwords.

Word vector representations enable machines to encode human language for spoken language understanding and processing. Confusion2vec, motivated from human speech production and perception, is a word vector representation which encodes ambiguities present in human spoken language in addition to semantics and syntactic information. Confusion2vec provides a robust spoken language representation by considering inherent human language ambiguities. In this paper, we propose a novel word vector space estimation by unsupervised learning on lattices output by an automatic speech recognition (ASR) system. We encode each word in Confusion2vec vector space by its constituent subword character n-grams. We show that the subword encoding helps better represent the acoustic perceptual ambiguities in human spoken language via information modeled on lattice-structured ASR output. The usefulness of the proposed Confusion2vec representation is evaluated using analogy and word similarity tasks designed for assessing semantic, syntactic and acoustic word relations. We also show the benefits of subword modeling for acoustic ambiguity representation on the task of spoken language intent detection. The results significantly outperform existing word vector representations when evaluated on erroneous ASR outputs, providing improvements up-to 13.12% relative to previous state-of-the-art in intent detection on ATIS benchmark dataset. We demonstrate that Confusion2vec subword modeling eliminates the need for retraining/adapting the natural language understanding models on ASR transcripts.

Degraded cortical temporal processing in the valproic acid-induced rat model of autism

Hearing disorders, such as abnormal speech perception, are frequently reported in individuals with autism. However, the mechanisms underlying these auditory-associated signature deficits in autism remain largely unknown. In this study, we documented significant behavioral impairments in the sound temporal rate discrimination task for rats prenatally exposed to valproic acid (VPA), a well-validated animal model for studying the pathology of autism. In parallel, there was a large-scale degradation in temporal information-processing in their primary auditory cortices (A1) at both levels of spiking outputs and synaptic inputs. Substantially increased spine density of excitatory neurons and decreased numbers of parvalbumin- and somatostatin-labeled inhibitory inter-neurons were also recorded in the A1 after VPA exposure. Given the fact that cortical temporal processing of sound is associated with speech perception in humans, these results in the animal model of VPA exposure provide insight into a possible neurological mechanism underlying auditory and language-related deficits in individuals with autism.

Modeling Unsupervised Empirical Adaptation by DPGMM and DPGMM-RNN Hybrid Model to Extract Perceptual Features for Low-Resource ASR

Speech feature extraction is critical for ASR systems. Such successful features as MFCC and PLP use filterbank techniques to model log-scaled speech perception but fail to model the adaptation of human speech perception by hearing experiences. Infant perception that is adapted by hearing speech without text may cause permanent brain state modifications (engrams) that serve as a physical fundamental basis for lifetime speech perception formation. This realization motivates us to propose to model such an unsupervised adaptation process, where adaptation denotes perception that is affected or changed by the history of experiences, with the Dirichlet Process Gaussian Mixture Model (DPGMM) and the DPGMM-RNN hybrid model to extract perceptual features to improve ASR. Our proposed features extend MFCC features with posteriorgrams extracted from the DPGMM algorithm or the DPGMM-RNN hybrid model. Our analysis shows that the DPGMM and DPGMM-RNN model perplexities agree with infant auditory perplexity to support that the proposed features are perceptual. Our ASR results verify the effectiveness of the proposed unsupervised features in such tasks as LVCSR on WSJ and ASR on noisy low-resource telephone conversations, compared with the supervised bottleneck features from Kaldi in ASR performance.

Frequency-Following Responses to Speech Sounds Are Highly Conserved across Species and Contain Cortical Contributions.

Time-varying pitch is a vital cue for human speech perception. Neural processing of time-varying pitch has been extensively assayed using scalp-recorded frequency-following responses (FFRs), an electrophysiological signal thought to reflect integrated phase-locked neural ensemble activity from subcortical auditory areas. Emerging evidence increasingly points to a putative contribution of auditory cortical ensembles to the scalp-recorded FFRs. However, the properties of cortical FFRs and precise characterization of laminar sources are still unclear. Here we used direct human intracortical recordings as well as extracranial and intracranial recordings from macaques and guinea pigs to characterize the properties of cortical sources of FFRs to time-varying pitch patterns. We found robust FFRs in the auditory cortex across all species. We leveraged representational similarity analysis as a translational bridge to characterize similarities between the human and animal models. Laminar recordings in animal models showed FFRs emerging primarily from the thalamorecipient layers of the auditory cortex. FFRs arising from these cortical sources significantly contributed to the scalp-recorded FFRs via volume conduction. Our research paves the way for a wide array of studies to investigate the role of cortical FFRs in auditory perception and plasticity.

Speech Computations of the Human Superior Temporal Gyrus.

Human speech perception results from neural computations that transform external acoustic speech signals into internal representations of words. The superior temporal gyrus (STG) contains the nonprimary auditory cortex and is a critical locus for phonological processing. Here, we describe how speech sound representation in the STG relies on fundamentally nonlinear and dynamical processes, such as categorization, normalization, contextual restoration, and the extraction of temporal structure. A spatial mosaic of local cortical sites on the STG exhibits complex auditory encoding for distinct acoustic-phonetic and prosodic features. We propose that as a population ensemble, these distributed patterns of neural activity give rise to abstract, higher-order phonemic and syllabic representations that support speech perception. This review presents a multi-scale, recurrent model of phonological processing in the STG, highlighting the critical interface between auditory and language systems.

How, when, and where predictions combine with speech in auditory cortexd

Human speech perception is rapid, accurate and robust to noise. These characteristics are enabled by top-down predictive computations that combine bottom-up sensory input with prior expectations. Here, I distinguish between two accounts of how predictions combine with speech signals in auditory cortex. Is expected speech content enhanced to leave a “sharpened” version of the sensory input? Or is expected content subtracted away to leave only those parts that are unexpected, i.e., “prediction error”? Based on fMRI and MEG data, I show that neural representations of spoken words are suppressed when listeners have strong predictions based on long-term lexical knowledge. This neural signature is uniquely consistent with the prediction error account, occurs in superior temporal cortex within 400 ms of speech input and in the delta/theta frequency band. These results provide a comprehensive window into predictive processing of speech, showing how, when and where predictions combine with sensory signals.

Automated detection of Glottal-related acoustic cues for feature-cue-based analysis

Detecting and interpreting individual acoustic cues to identify features that distinguish among speech sounds is thought to play a key role in automatic speech processing, modeling human speech perception, detecting and diagnosing speech disabilities (and tracking the effects of treatment for those disabilities), and studying cross-language differences. The individual acoustic cues form a bridge between transcription using phonemic symbols (phonemes) and raw acoustic measurements from the signal and allow the analysis to capture systematic language-related differences in how words are pronounced in different contexts–differences that are critical to speech recognition. Within this framework, analysis of glottal-related acoustic cues, such as aspiration and voicing, can contribute to the identification of phonemes that exhibit these acoustic cues in their respective productions. In this study, we propose an algorithm that uses time-series data processing methods and machine learning models to accurately predict the presence of the glottal cue labels from waveform analysis. We further highlight the accuracy of our model and cue-based labeling method in identifying speech modifications over phone and phoneme-based methods.