Mechanisms Of Speech Perception Research Articles

Evaluating speech-in-speech perception via a humanoid robot.

Underlying mechanisms of speech perception masked by background speakers, a common daily listening condition, are often investigated using various and lengthy psychophysical tests. The presence of a social agent, such as an interactive humanoid NAO robot, may help maintain engagement and attention. However, such robots potentially have limited sound quality or processing speed. As a first step toward the use of NAO in psychophysical testing of speech- in-speech perception, we compared normal-hearing young adults' performance when using the standard computer interface to that when using a NAO robot to introduce the test and present all corresponding stimuli. Target sentences were presented with colour and number keywords in the presence of competing masker speech at varying target-to-masker ratios. Sentences were produced by the same speaker, but voice differences between the target and masker were introduced using speech synthesis methods. To assess test performance, speech intelligibility and data collection duration were compared between the computer and NAO setups. Human-robot interaction was assessed using the Negative Attitude Toward Robot Scale (NARS) and quantification of behavioural cues (backchannels). Speech intelligibility results showed functional similarity between the computer and NAO setups. Data collection durations were longer when using NAO. NARS results showed participants had a relatively positive attitude toward "situations of interactions" with robots prior to the experiment, but otherwise showed neutral attitudes toward the "social influence" of and "emotions in interaction" with robots. The presence of more positive backchannels when using NAO suggest higher engagement with the robot in comparison to the computer. Overall, the study presents the potential of the NAO for presenting speech materials and collecting psychophysical measurements for speech-in-speech perception.

Psycholinguistics, Corpus Linguistics, and Text Analytics: Potential for Interaction

This article explores the possibilities of interaction between psycholinguistics, corpus linguistics, and text mining. When it comes to big data processing, the recent advances in psycholinguistics must be taken into account. In our digital reality, we should no longer view text as a mere sum of individual language units that are assembled based on their similarities in meaning. The insights of psycholinguistics into text have been revised, as in-depth text analysis unravels the new roles of the sender and the recipient, bringing to the fore qualitative methods for processing large data arrays. Without studying the mechanisms of speech generation and perception, it is impossible to improve language neural networks and machine translation engines. Here, the potential of psycholinguistics for training specialists in text analytics is discussed. As an example, the advantages of integrating the disciplines “Psycholinguistics” and “Psycholinguistic Methods for Studying Language and Text” into the psychological and linguistic module of the Master’s program “Text Analytics in Education and Science”, which has been launched at Kazan Federal University (Russia), are shown. The psycholinguistic component of the discipline “Corpus Linguistics” for Master’s students is described.

Words Listening Related Electroencephalography Spectrum Perturbations in Normally Developing Children and Sensory Alalia Children

The analysis of electroencephalography (EEG) event related spectrum perturbations in 23 normally developing and 23 sensory alalia children of age between 4 and 10 years old during listening to nouns is conducted. In EEG θ-frequency range of healthy children, synchronization was detected in the frontal, central, and temporal leads, and desynchronization was found in most leads in children with sensory alalia. Intergroup differences in the reactivity of the EEG θ-range reached the level of statistical significance in C4 lead. In the α‑range, EEG desynchronization was observed in typically developing children, and synchronization was observed in children with sensory alalia. Differences in the α-activity power reached the level of statistical significance in leads C3, F3, P3, O1. An EEG θ-band synchronization decrease during words listening in children with sensory alalia may reflect an impairment of retrieving words from memory and less emotional involvement in the speech perception process, while the absence of α-activity desynchronization in the central leads is a impairment of the functioning of sensorimotor neural networks, involved in the speech perception and generation. The revealed features of EEG reactivity are important for understanding the central speech perception mechanisms in normal children, as well as in children with disorders such as sensory alalia.

Vowel speech recognition from rat electroencephalography using long short-term memory neural network.

Over the years, considerable research has been conducted to investigate the mechanisms of speech perception and recognition. Electroencephalography (EEG) is a powerful tool for identifying brain activity; therefore, it has been widely used to determine the neural basis of speech recognition. In particular, for the classification of speech recognition, deep learning-based approaches are in the spotlight because they can automatically learn and extract representative features through end-to-end learning. This study aimed to identify particular components that are potentially related to phoneme representation in the rat brain and to discriminate brain activity for each vowel stimulus on a single-trial basis using a bidirectional long short-term memory (BiLSTM) network and classical machine learning methods. Nineteen male Sprague-Dawley rats subjected to microelectrode implantation surgery to record EEG signals from the bilateral anterior auditory fields were used. Five different vowel speech stimuli were chosen, /a/, /e/, /i/, /o/, and /u/, which have highly different formant frequencies. EEG recorded under randomly given vowel stimuli was minimally preprocessed and normalized by a z-score transformation to be used as input for the classification of speech recognition. The BiLSTM network showed the best performance among the classifiers by achieving an overall accuracy, f1-score, and Cohen’s κ values of 75.18%, 0.75, and 0.68, respectively, using a 10-fold cross-validation approach. These results indicate that LSTM layers can effectively model sequential data, such as EEG; hence, informative features can be derived through BiLSTM trained with end-to-end learning without any additional hand-crafted feature extraction methods.

Temporal fine structure influences voicing confusions for consonant identification in multi-talker babble.

To understand the mechanisms of speech perception in everyday listening environments, it is important to elucidate the relative contributions of different acoustic cues in transmitting phonetic content. Previous studies suggest that the envelope of speech in different frequency bands conveys most speech content, while the temporal fine structure (TFS) can aid in segregating target speech from background noise. However, the role of TFS in conveying phonetic content beyond what envelopes convey for intact speech in complex acoustic scenes is poorly understood. The present study addressed this question using online psychophysical experiments to measure the identification of consonants in multi-talker babble for intelligibility-matched intact and 64-channel envelope-vocoded stimuli. Consonant confusion patterns revealed that listeners had a greater tendency in the vocoded (versus intact) condition to be biased toward reporting that they heard an unvoiced consonant, despite envelope and place cues being largely preserved. This result was replicated when babble instances were varied across independent experiments, suggesting that TFS conveys voicing information beyond what is conveyed by envelopes for intact speech in babble. Given that multi-talker babble is a masker that is ubiquitous in everyday environments, this finding has implications for the design of assistive listening devices such as cochlear implants.

A Literature Review of Neural Tracking to Speech Envelope from the View of Audiology

Despite the significant contribution of hearing assistive devices, medications, and surgery to restoring auditory periphery, a large number of people with hearing loss still struggle with understanding speech. This leads many studies on speech perception to move towards the central auditory functions by looking at associated brain activities using macroscopic recording tools such as electroencephalography (EEG). Up until a few years ago, however, limitation has been given to the brain scientists who attempted to investigate speech perception mechanisms using the EEG. In particular, short duration of speech segments has inevitably been used to elicit auditory evoked potential, even though they were too brief to be considered as speech. Today, however, advance in neural engineering and better understanding of neural mechanism have better facilitated brain scientists to perform studies with running stream of continuous speech and expand the scope of EEG studies to include comprehension of more realistic speech envelope. The purpose of this study is to review literatures on neural tracking to speech envelope to discuss it in Audiology perspective. This review article consists of seven subjects including introduction, neural tracking theories, neural tracking measure, signal processing & analysis, literature review on neural tracking associated with hearing loss, application of neural tracking to audiology, and conclusion. We noted that neural tracking has potential to be used in clinical sets to objectively evaluate speech comprehension for people with hearing loss in the future.

Human EEG and Recurrent Neural Networks Exhibit Common Temporal Dynamics During Speech Recognition.

Recent deep-learning artificial neural networks have shown remarkable success in recognizing natural human speech, however the reasons for their success are not entirely understood. Success of these methods might be because state-of-the-art networks use recurrent layers or dilated convolutional layers that enable the network to use a time-dependent feature space. The importance of time-dependent features in human cortical mechanisms of speech perception, measured by electroencephalography (EEG) and magnetoencephalography (MEG), have also been of particular recent interest. It is possible that recurrent neural networks (RNNs) achieve their success by emulating aspects of cortical dynamics, albeit through very different computational mechanisms. In that case, we should observe commonalities in the temporal dynamics of deep-learning models, particularly in recurrent layers, and brain electrical activity (EEG) during speech perception. We explored this prediction by presenting the same sentences to both human listeners and the Deep Speech RNN and considered the temporal dynamics of the EEG and RNN units for identical sentences. We tested whether the recently discovered phenomenon of envelope phase tracking in the human EEG is also evident in RNN hidden layers. We furthermore predicted that the clustering of dissimilarity between model representations of pairs of stimuli would be similar in both RNN and EEG dynamics. We found that the dynamics of both the recurrent layer of the network and human EEG signals exhibit envelope phase tracking with similar time lags. We also computed the representational distance matrices (RDMs) of brain and network responses to speech stimuli. The model RDMs became more similar to the brain RDM when going from early network layers to later ones, and eventually peaked at the recurrent layer. These results suggest that the Deep Speech RNN captures a representation of temporal features of speech in a manner similar to human brain.

An effectively causal deep learning algorithm to increase intelligibility in untrained noises for hearing-impaired listeners

Real-time operation is critical for noise reduction in hearing technology. The essential requirement of real-time operation is causality-that an algorithm does not use future time-frame information and, instead, completes its operation by the end of the current time frame. This requirement is extended currently through the concept of "effectively causal," in which future time-frame information within the brief delay tolerance of the human speech-perception mechanism is used. Effectively causal deep learning was used to separate speech from background noise and improve intelligibility for hearing-impaired listeners. A single-microphone, gated convolutional recurrent network was used to perform complex spectral mapping. By estimating both the real and imaginary parts of the noise-free speech, both the magnitude and phase of the estimated noise-free speech were obtained. The deep neural network was trained using a large set of noises and tested using complex noises not employed during training. Significant algorithm benefit was observed in every condition, which was largest for those with the greatest hearing loss. Allowable delays across different communication settings are reviewed and assessed. The current work demonstrates that effectively causal deep learning can significantly improve intelligibility for one of the largest populations of need in challenging conditions involving untrained background noises.

Toward a new model for speech perception: the Universal Perceptual Model (UPM) of second language.

Several speech models have been formed in the past aiming to predict the abilities of nonnative listeners or learners in perceiving and producing speech sounds. The present paper proposes a new model for speech perception, the Universal Perceptual Model of Second Language (henceforth, UPM). UPM assumes that second language phone acquisition is strongly affected by the speakers' native language but still the window of phone learning is open due to the universality of speech sounds. Also, it supports that second language phones are initially activated as disoriented phonetic units. In this paper, we provide some initial insights into the predictability of the model. UPM uses degrees of overlap and chance criteria to form its predictions. We recruited Cypriot Greek novice learners of Italian who participated in two psychoacoustic tasks in which they classified and discriminated Italian vowels, respectively. The findings demonstrated that the degree of overlap between two nonnative phones may be a good predictor of the speakers' discrimination accuracy over these phones. UPM might be a useful model which aims to better explain speech perception mechanisms and patterns of speech acquisition.

Contrasting behavioral looking procedures: a case study on infant speech segmentation

This paper compared three different procedures common in infant speech perception research: a headturn preference procedure (HPP) and a central-fixation (CF) procedure with either automated eye-tracking (CF-ET) or manual coding (CF-M). In theory, such procedures all measure the same underlying speech perception and learning mechanisms and the choice between them should ideally be irrelevant in unveiling infant preference. However, the ManyBabies study (ManyBabies Consortium, 2019), a cross-laboratory collaboration on infants’ preference for child-directed speech, revealed that choice of procedure can modulate effect sizes. Here we examined whether procedure also modulates preference in paradigms that add a learning phase prior to test: a speech segmentation paradigm. Such paradigms are particularly important for studying the learning mechanisms infants can employ for language acquisition. We carried out the same familiarization-then-test experiment with the three different procedures (32 unique infants per procedure). Procedures were compared on various factors, such as overall effect, average looking time and drop-out rate. The key observations are that the HPP yielded a larger familiarity preference, but also reported larger drop-out rates. This raises questions about the generalizability of results. We argue that more collaborative research into different procedures in infant preference experiments is required in order to interpret the variation in infant preferences more accurately.

СУЩЕСТВУЕТ ЛИ СЛОВО В ТЕКСТЕ: ПСИХОЛОГИЧЕСКИЕ И ЛИНГВИСТИЧЕСКИЕ АСПЕКТЫ СООТНОШЕНИЯ ПЕРЦЕПТИВНОЙ И АРТИКУЛЯЦИОННОЙ БАЗ ЯЗЫКА

Complicated analytic and synthetic mechanisms of speech signal processing underlie the basis for listening comprehension skills, therefore, while teaching listening skills, the mechanisms of oral foreign speech perception remain crucial and properly addressed. The article looks into the formation of the perceptual language base while perceiving German as L2 speech. In this paper, we discuss a number of issues including the priority role of listening comprehension, mechanisms of perception and the interaction of the articulatory and perceptual language bases. The word is viewed as the basic unit of perception. The listening mechanisms of the German word are described in terms of significant linguistic features. The necessity of purposeful formation of basic oral speech perception skills is shown. The operating units of such formation are sets of keywords in the unity of their sound patterns and meanings, the latter characterized by the perception relevant features. The techniques of operating these features in the designed coursebooks are also observed.

Mechanisms of Localization and Speech Perception with Colocated and Spatially Separated Noise and Speech Maskers Under Single-Sided Deafness with a Cochlear Implant.

This study tested listeners with a cochlear implant (CI) in one ear and acoustic hearing in the other ear, to assess their ability to localize sound and to understand speech in collocated or spatially separated noise or speech maskers. Eight CI listeners with contralateral acoustic hearing ranging from normal hearing to moderate sensorineural hearing loss were tested. Localization accuracy was measured in five of the listeners using stimuli that emphasized the separate contributions of interaural level differences (ILDs) and interaural time differences (ITD) in the temporal envelope and/or fine structure. Sentence recognition was tested in all eight CI listeners, using collocated and spatially separated speech-shaped Gaussian noise and two-talker babble. Performance was compared with that of age-matched normal-hearing listeners via loudspeakers or via headphones with vocoder simulations of CI processing. Localization improved with the CI but only when high-frequency ILDs were available. Listeners experienced no additional benefit via ITDs in the stimulus envelope or fine structure using real or vocoder-simulated CIs. Speech recognition in two-talker babble improved with a CI in seven of the eight listeners when the target was located at the front and the babble was presented on the side of the acoustic-hearing ear, but otherwise showed little or no benefit of a CI. Sound localization can be improved with a CI in cases of significant residual hearing in the contralateral ear, but only for sounds with high-frequency content, and only based on ILDs. In speech understanding, the CI contributed most when it was in the ear with the better signal to noise ratio with a speech masker.

Neural tracking of the speech envelope in cochlear implant users

Objective. When listening to speech, the brain tracks the speech envelope. It is possible to reconstruct this envelope from EEG recordings. However, in people who hear using a cochlear implant (CI), the artifacts caused by electrical stimulation of the auditory nerve contaminate the EEG. The objective of this study is to develop and validate a method for assessing the neural tracking of speech envelope in CI users. Approach. To obtain EEG recordings free of stimulus artifacts, the electrical stimulation is periodically interrupted. During these stimulation gaps, artifact-free EEG can be sampled and used to train a linear envelope decoder. EEG recordings obtained during audible and inaudible (i.e. sub-threshold) stimulation were used to characterize the artifacts and their influence on the envelope reconstruction. Main results. The present study demonstrates for the first time that neural tracking of the speech envelope can be measured in response to ongoing electrical stimulation. The responses were validated to be truly neural and not affected by stimulus artifact. Significance. Besides applications in audiology and neuroscience, the characterization and elimination of stimulus artifacts will enable future EEG studies involving continuous speech in CI users. Measures of neural tracking of the speech envelope reflect interesting properties of the listener’s perception of speech, such as speech intelligibility or attentional state. Successful decoding of neural envelope tracking will open new possibilities to investigate the neural mechanisms of speech perception with a CI.

Cognitive Speech Coding: Examining the Impact of Cognitive Speech Processing on Speech Compression

Speech coding is a field in which compression paradigms have not changed in the last 30 years. Speech signals are most commonly encoded with compression methods that have roots in linear predictive theory dating back to the early 1940s. This article bridges this influential theory with recent cognitive studies applicable in speech communication engineering. It reviews the mechanisms of speech perception that have led to perceptual speech coding. The focus is on human speech communication and machine learning and the application of cognitive speech processing in speech compression that presents a paradigm shift from perceptual (auditory) speech processing toward cognitive (auditory plus cortical) speech processing.

The right hemisphere is highlighted in connected natural speech production and perception

Current understanding of the cortical mechanisms of speech perception and production stems mostly from studies that focus on single words or sentences. However, it has been suggested that processing of real-life connected speech may rely on additional cortical mechanisms. In the present study, we examined the neural substrates of natural speech production and perception with magnetoencephalography by modulating three central features related to speech: amount of linguistic content, speaking rate and social relevance. The amount of linguistic content was modulated by contrasting natural speech production and perception to speech-like non-linguistic tasks. Meaningful speech was produced and perceived at three speaking rates: normal, slow and fast. Social relevance was probed by having participants attend to speech produced by themselves and an unknown person. These speech-related features were each associated with distinct spatiospectral modulation patterns that involved cortical regions in both hemispheres. Natural speech processing markedly engaged the right hemisphere in addition to the left. In particular, the right temporo-parietal junction, previously linked to attentional processes and social cognition, was highlighted in the task modulations. The present findings suggest that its functional role extends to active generation and perception of meaningful, socially relevant speech.

Perceptual Associations Between Words and Speaker Age

Recognition proficiency was measured in a lexical decision experiment using Korean words including ones that are stereotypically associated with younger or older people. Both accuracy rates and response times were significantly improved when a word was spoken in a voice whose age matched the age category of the word based on stereotype-based age rating. Accuracy rates were also significantly improved when a word was spoken in a voice whose age matched the frequency-based age category, although the effect on response times was marginal. The results are discussed in the context of an exemplar-based account incorporating social exemplar activation. First, the activation of social information is influenced by distributional properties of lexical use across social groups, indicating that social information of speakers is not only linked to sub-lexical representations as shown in previous studies, but also directly linked to lexical representations through social indices. Furthermore, the presence of an additional effect of stereotypes expands our understanding of the speech perception mechanism, suggesting that the social indices are enhanced by stereotypical associations between words and age categories. When listening to words with a wide spectrum of age-related variability, word-stereotypes activate social information, guiding to the lexical representations via social indices.

ПРИМЕНЕНИЕ ЧАСТОТНОГО МАСКИРОВАНИЯ ПРИ MFCC-ПАРАМЕТРИЗАЦИИ РЕЧИ НА ФОНЕ ШУМОВ

Introduction: MFCCs are widely used in speech signals parameterization, however their effectiveness significantly decreases when the signal contains a noise term. The work proposes and studies a modification of the traditional MFCC calculation algorithm by introducing additional signal transformations based on the mechanisms of speech production and perception. Results: We propose a psychoacoustic model which allows you to take into account the frequency-masking effect in human auditory perception. In addition, considering how formant areas are formed in the voice spectrum, we propose to influence the spectral counts corresponding to multiple harmonics of the fundamental tone. The modified algorithm was tested in a single-word recognition system adapted for speech signal parameterization with only MFCCs. We demonstrated a positive effect of using the proposed additional speech signal transformations in the parameterization algorithm. Practical relevance: The proposed approach to MFCC calculation for the speech signal segment allows you to improve MFCC usage effectiveness in a variety of speech applications.

Enhancement of brain event-related potentials to speech sounds is associated with compensated reading skills in dyslexic children with familial risk for dyslexia

Specific reading disability, dyslexia, is a prevalent and heritable disorder impairing reading acquisition characterized by a phonological deficit. However, the underlying mechanism of how the impaired phonological processing mediates resulting dyslexia or reading disabilities remains still unclear. Using ERPs we studied speech sound processing of 30 dyslexic children with familial risk for dyslexia, 51 typically reading children with familial risk for dyslexia, and 58 typically reading control children. We found enhanced brain responses to shortening of a phonemic length in pseudo-words (/at:a/ vs. /ata/) in dyslexic children with familial risk as compared to other groups. The enhanced brain responses were associated with better performance in behavioral phonemic length discrimination task, as well as with better reading and writing accuracy. Source analyses revealed that the brain responses of sub-group of dyslexic children with largest responses originated from a more posterior area of the right temporal cortex as compared to the responses of the other participants. This is the first electrophysiological evidence for a possible compensatory speech perception mechanism in dyslexia. The best readers within the dyslexic group have probably developed alternative strategies which employ compensatory mechanisms substituting their possible earlier deficit in phonological processing and might therefore be able to perform better in phonemic length discrimination and reading and writing accuracy tasks. However, we speculate that for reading fluency compensatory mechanisms are not that easily built and dyslexic children remain slow readers during their adult life.

Theta–gamma coupling reflects the interaction of bottom-up and top-down processes in speech perception in children

This study investigates how the interaction of different brain oscillations (particularly theta–gamma coupling) modulates the bottom-up and top-down processes during speech perception. We employed a speech perception paradigm that manipulated the congruency between a visually presented picture and an auditory stimulus and asked participants to judge whether they matched or mismatched. A group of children (mean age 10years, 5months) participated in this study and their electroencephalographic (EEG) data were recorded while performing the experimental task. It was found that in comparison with mismatch condition, match condition facilitated speech perception by eliciting greater theta–gamma coupling in the frontal area and smaller theta–gamma coupling in the left temporal area. These findings suggested that a top-down facilitation effect from congruent visual pictures engaged different mechanisms in low-level sensory (temporal) regions and high-level linguistic and decision (frontal) regions. Interestingly, hemispheric asymmetry is with higher theta–gamma coupling in the match condition in the right hemisphere and higher theta–gamma coupling in the mismatch condition in the left hemisphere. This indicates that a fast global processing strategy and a slow detailed processing strategy were differentially adopted in the match and mismatch conditions. This study provides new insight into the mechanisms of speech perception from the interaction of different oscillatory activities and provides neural evidence for theories of speech perception allowing for top-down feedback connections. Furthermore, it sheds light on children's speech perception development by showing a similar pattern of integration of bottom-up and top-down information during speech perception as previous studies have revealed in adults.

Imitation of a talker improves perception of the talker's speech

Human perceivers have a tendency to imitate the idiolect (talker-specific articulatory style) of a perceived talker. This phonetic convergence manifests between perceivers during live conversations and when perceivers shadow (say aloud) the speech spoken by a pre-recorded talker. The following investigation explores the potential facilitative effects of phonetic convergence on subsequent perception of the imitated talker's speech. To test this, the strength of phonetic convergence was manipulated by varying the delay of perceivers' shadowing responses. When perceivers are required to delay their shadowing responses, phonetic convergence has been found to reduce [Goldinger, Psychol. Rev. 105(2), 215–279 (1998)]. If phonetic convergence can facilitate later perception of speech spoken by the shadowed talker, then perceivers who immediately shadow a talker should better identify the speech spoken by that talker, compared to perceivers who delayed their shadowing responses. Results suggest immediate—but not delayed—shadowing facilitates later identification of shadowed words spoken by the shadowed talker, compared to a non-shadowed talker. However, the facilitation effect does not transfer to utterances of novel (non-shadowed) words. The results may suggest a link between the mechanisms of speech perception and speech production based on idiolectic information available within lexical episodes.