Human Speech Perception Research Articles

Advances in Automatic Speech Recognition: From Audio-Only To Audio-Visual Speech Recognition

Major developments in the field of finding more natural ways of interacting with computers have been taking place. The clear focus lies on making technology more approachable to people. The concept that computers can comprehend our various gestures by eyes, voices, touch and our different movements to interact is called the Natural User Interface (NUI). Today, many of these elements are available in mobile phones, PCs, and in other devices. Speech technologies particularly play a substantial role in this evolution. Significant advancement in automatic speech recognition (ASR) for well defined applications like dictation and medium vocabulary transaction processing assignments in comparatively controlled environments have been made. But, automatic speech recognition still has to reach a level needed for speech to become a completely pervasive user interface because even in clean acoustic surroundings, the state of ASR system performance falls behind human speech perception. Visual speech recognition, however, is a promising source of extra speech information and it has successfully exhibited to enhance noise robustness of automatic speech recognizers, thereby promising to expand their usability in the human computer interaction. In this paper, the main components of audio-visual speech recognition, i.e., the audio and the video components are discussed, along with the latest advancements made in this field. Further, the research goes beyond the recent advancements and discusses the future scope of audio video speech recognition and mentions some likely future developments, evaluating each on the basis of its performance. Graphs are plotted based on experiments to depict the performance improvements from audio- only ASR to audio-video ASR, along with its expected performance level in future.

Technology in Phonetic Science: Setting Up a Basic Phonetics Laboratory

Abstract The present paper discusses some techniques and procedures used in experimental phonetic research. The use of technology and its recent widespread availability have been instrumental in our developing a better understanding of the basic processes of human speech production and perception. Specifically, this paper reviews some of the uses of the software package Praat for acoustic and perceptual research, as well as software products such as R, Akustyk, SuperLab, and E-Prime, among others. Finally, we offer some remarks on the basic equipment (hardware) one would need to purchase in order to carry out modern experimental phonetic research in the field as well as in the laboratory.

Zebra finches and Dutch adults exhibit the same cue weighting bias in vowel perception

Vocal tract resonances, called formants, are the most important parameters in human speech production and perception. They encode linguistic meaning and have been shown to be perceived by a wide range of species. Songbirds are also sensitive to different formant patterns in human speech. They can categorize words differing only in their vowels based on the formant patterns independent of speaker identity in a way comparable to humans. These results indicate that speech perception mechanisms are more similar between songbirds and humans than realized before. One of the major questions regarding formant perception concerns the weighting of different formants in the speech signal (“acoustic cue weighting”) and whether this process is unique to humans. Using an operant Go/NoGo design, we trained zebra finches to discriminate syllables, whose vowels differed in their first three formants. When subsequently tested with novel vowels, similar in either their first formant or their second and third formants to the familiar vowels, similarity in the higher formants was weighted much more strongly than similarity in the lower formant. Thus, zebra finches indeed exhibit a cue weighting bias. Interestingly, we also found that Dutch speakers when tested with the same paradigm exhibit the same cue weighting bias. This, together with earlier findings, supports the hypothesis that human speech evolution might have exploited general properties of the vertebrate auditory system.

Speech Perception: A Language-Trained Chimpanzee Weighs In

A language-trained chimpanzee is able to interpret synthetic 'auditory caricatures' as speech. Important components of human speech perception thus rely upon general auditory mechanisms that predated the evolution of spoken language.

Bird Speech Perception and Vocal Production: A Comparison with Humans

Research into speech perception by nonhuman animals can be crucially informative in assessing whether specific perceptual phenomena in humans have evolved to decode speech, or reflect more general traits. Birds share with humans not only the capacity to use complex vocalizations for communication but also many characteristics of its underlying developmental and mechanistic processes; thus, birds are a particularly interesting group for comparative study. This review first discusses commonalities between birds and humans in perception of speech sounds. Several psychoacoustic studies have shown striking parallels in seemingly speech-specific perceptual phenomena, such as categorical perception of voice-onset-time variation, categorization of consonants that lack phonetic invariance, and compensation for coarticulation. Such findings are often regarded as evidence for the idea that the objects of human speech perception are auditory or acoustic events rather than articulations. Next, I highlight recent research on the production side of avian communication that has revealed the existence of vocal tract filtering and articulation in bird species-specific vocalization, which has traditionally been considered a hallmark of human speech production. Together, findings in birds show that many of characteristics of human speech perception are not uniquely human but also that a comparative approach to the question of what are the objects of perception—articulatory or auditory events—requires careful consideration of species-specific vocal production mechanisms.

Bird Speech Perception and Vocal Production: A Comparison with Humans

Research into speech perception by nonhuman animals can be crucially informative in assessing whether specific perceptual phenomena in humans have evolved to decode speech, or reflect more general traits. Birds share with humans not only the capacity to use complex vocalizations for communication but also many characteristics of its underlying developmental and mechanistic processes; thus, birds are a particularly interesting group for comparative study. This review first discusses commonalities between birds and humans in perception of speech sounds. Several psychoacoustic studies have shown striking parallels in seemingly speech-specific perceptual phenomena, such as categorical perception of voice-onset-time variation, categorization of consonants that lack phonetic invariance, and compensation for coarticulation. Such findings are often regarded as evidence for the idea that the objects of human speech perception are auditory or acoustic events rather than articulations. Next, I highlight recent research on the production side of avian communication that has revealed the existence of vocal tract filtering and articulation in bird species-specific vocalization, which has traditionally been considered a hallmark of human speech production. Together, findings in birds show that many of characteristics of human speech perception are not uniquely human but also that a comparative approach to the question of what are the objects of perception--articulatory or auditory events--requires careful consideration of species-specific vocal production mechanisms.

Extracting amplitude modulations from speech in the time domain

Natural sounds can be characterised by patterns of changes in loudness (amplitude modulations), and human speech perception studies have focused on the low frequencies contained in the gross temporal structure of speech. Low-pass filtering the temporal envelopes of sub-band filtered speech maintains intelligibility, but it remains unclear how the human auditory system could perform such a modulation domain analysis or even if it does so at all. It is difficult to further manipulate amplitude modulations through frequency-domain filtering to investigate cues the system may use. The current work focuses on a time-domain decomposition of filter output envelopes into pulses of amplitude modulation. The technique demonstrates that signals low-pass filtered in the modulation domain maintain bursts of energy which are comparable to those that can be extracted entirely within the time-domain. This paper presents preliminary work that suggests a time-domain approach, which focuses on the instantaneous features of transient changes in loudness, can be used to study the content of human speech. This approach should be pursued as it allows human speech intelligibility mechanisms to be investigated from a new perspective.

Evaluation of two algorithms for detecting human frequency-following responses to voice pitch

Objective. Voice pitch carries important cues for speech perception in humans. Recent studies have shown the feasibility of recording the frequency-following response (FFR) to voice pitch in normal-hearing listeners. The presence of such a response, however, has been dependent on subjective interpretation of experimenters. The purpose of this study was to develop and test an automated procedure including a control-experimental protocol and response-threshold criteria suitable for extracting FFRs to voice pitch, and compare the results to human judgments. Design. A set of four Mandarin tones (Tone 1 flat; Tone 2 rising; Tone 3 dipping; and Tone 4 falling) were prepared to reflect the four contrastive pitch contours. Two distinctive algorithms, short-term autocorrelation in the time domain and narrow-band spectrogram in the frequency domain, were used to estimate the Frequency Error, Slope Error, Tracking Accuracy, Pitch Strength and Pitch-Noise Ratio of the recordings from individual listeners as well as the power and false-positive rates of each algorithm. Study Sample. Eleven native speakers (five males; age: mean ± SD = 31.4 ± 4.7 years) of Mandarin Chinese were recruited. Results. The results demonstrated that both algorithms were suitable for extracting FFRs and the objective measures showed comparable results to human judgments. Conclusions. The automated procedure used in this study, including the use of the control-experimental protocol and response thresholds used for each of the five objective indices, can be used for difficult-to-test patients and may prove to be useful as an assessment and diagnostic method in both clinical and basic research efforts.SumarioLa altura tonal de la voz contiene importantes claves para la percepción del habla en humanos. Estudios recientes mostraron la factibilidad de registrar la respuesta de seguimiento a la frecuencia (FFR), de la frecuencia vocal en normo-oyentes. No obstante, la presencia de esta respuesta ha sido dependiente de la interpretación subjetiva de los experimentadores. El propósito de este estudio fue desarrollar y probar un procedimiento automatizado, incluyendo un protocolo de control experimental y criterios de respuesta-umbral, apropiados para la extracción de FFRs de la frecuencia vocal y comparar los resultados con los juicios humanos. Se preparó un set de tonos del Mandarín (tono 1, plano; tono 2, ascendente; tono, 3 silbado y tono 4, descendente), para mostrar los cuatro contornos de frecuencia contrastante. Se usaron dos algoritmos distintivos, de auto-correlación a corto plazo, para estimar el error en la frecuencia y en el gradiente, la exactitud del rastreo, la fortaleza de la frecuencia y la relación tono-ruido de los registros de los oyentes individuales así como de la fuerza y las tasas de falsos positivos de cada algoritmo. Los resultados demostraron que ambos algoritmos fueron apropiados para extraer las FFRs y que las mediciones objetivas mostraron resultados comparables a los de los juicios humanos.

Human phoneme recognition depending on speech-intrinsic variability

The influence of different sources of speech-intrinsic variation (speaking rate, effort, style and dialect or accent) on human speech perception was investigated. In listening experiments with 16 listeners, confusions of consonant-vowel-consonant (CVC) and vowel-consonant-vowel (VCV) sounds in speech-weighted noise were analyzed. Experiments were based on the OLLO logatome speech database, which was designed for a man-machine comparison. It contains utterances spoken by 50 speakers from five dialect/accent regions and covers several intrinsic variations. By comparing results depending on intrinsic and extrinsic variations (i.e., different levels of masking noise), the degradation induced by variabilities can be expressed in terms of the SNR. The spectral level distance between the respective speech segment and the long-term spectrum of the masking noise was found to be a good predictor for recognition rates, while phoneme confusions were influenced by the distance to spectrally close phonemes. An analysis based on transmitted information of articulatory features showed that voicing and manner of articulation are comparatively robust cues in the presence of intrinsic variations, whereas the coding of place is more degraded. The database and detailed results have been made available for comparisons between human speech recognition (HSR) and automatic speech recognizers (ASR).

Responsiveness of the human auditory cortex to degraded speech sounds: Reduction of amplitude resolution vs. additive noise

The cortical mechanisms underlying human speech perception in acoustically adverse conditions remain largely unknown. Besides distortions from external sources, degradation of the acoustic structure of the sound itself poses further demands on perceptual mechanisms. We conducted a magnetoencephalography (MEG) study to reveal whether the perceptual differences between these distortions are reflected in cortically generated auditory evoked fields (AEFs). To mimic the degradation of the internal structure of sound and external distortion, we degraded speech sounds by reducing the amplitude resolution of the signal waveform and by using additive noise, respectively. Since both distortion types increase the relative strength of high frequencies in the signal spectrum, we also used versions of the stimuli which were low-pass filtered to match the tilted spectral envelope of the undistorted speech sound. This enabled us to examine whether the changes in the overall spectral shape of the stimuli affect the AEFs. We found that the auditory N1m response was substantially enhanced as the amplitude resolution was reduced. In contrast, the N1m was insensitive to distorted speech with additive noise. Changing the spectral envelope had no effect on the N1m. We propose that the observed amplitude enhancements are due to an increase in noisy spectral harmonics produced by the reduction of the amplitude resolution, which activates the periodicity-sensitive neuronal populations participating in pitch extraction processes. The current findings suggest that the auditory cortex processes speech sounds in a differential manner when the internal structure of sound is degraded compared with the speech distorted by external noise.

Intelligibility predictors and neural representation of speech

Intelligibility predictors tell us a great deal about human speech perception, in particular which acoustic factors strongly effect human behavior, and which do not. A particular intelligibility predictor, the Articulation Index (AI), is interesting because it models human behavior in noise, and its form has implications about representation of speech in the brain. Specifically, the Articulation Index implies that a listener pre-consciously estimates the masking noise distribution and uses it to classify time/frequency samples as speech or non-speech. We classify consonants using representations of speech and noise which are consistent with this hypothesis, and determine whether their error rate and error patterns are more or less consistent with human behavior than representations typical of automatic speech recognition systems. The new representations resulted in error patterns more similar to humans in cases where the testing and training data sets do not have the same masking noise spectrum.

Automatic speech emotion recognition using modulation spectral features

In this study, modulation spectral features (MSFs) are proposed for the automatic recognition of human affective information from speech. The features are extracted from an auditory-inspired long-term spectro-temporal representation. Obtained using an auditory filterbank and a modulation filterbank for speech analysis, the representation captures both acoustic frequency and temporal modulation frequency components, thereby conveying information that is important for human speech perception but missing from conventional short-term spectral features. On an experiment assessing classification of discrete emotion categories, the MSFs show promising performance in comparison with features that are based on mel-frequency cepstral coefficients and perceptual linear prediction coefficients, two commonly used short-term spectral representations. The MSFs further render a substantial improvement in recognition performance when used to augment prosodic features, which have been extensively used for emotion recognition. Using both types of features, an overall recognition rate of 91.6% is obtained for classifying seven emotion categories. Moreover, in an experiment assessing recognition of continuous emotions, the proposed features in combination with prosodic features attain estimation performance comparable to human evaluation.

Frequency-Lowering Hearing Aids: Verification Tools and Research Needs

Frequency-Lowering Hearing Aids: Verification Tools and Research Needs

Two-scale Auditory Feature Based Non-intrusive Speech Quality Evaluation

This paper proposes a novel two-scale auditory feature based algorithm for non-intrusive evaluation of speech quality. The neuron firing probabilities along the length of the basilar membrane, from an explicit auditory model, are used to extract features from the distorted speech signal. This is in contrast to previous methods, which either use standard vocal tract based features, or incorporate only some aspects of the human auditory perception mechanism. The features are extracted at two scales, namely a global scale spanning all voiced frames in an utterance, and a local scale spanning voiced frames from contiguous voiced segments in the utterance. This is followed by a simple information fusion at the score level using Gaussian Mixture Models (GMMs). The use of an explicit auditory model to extract features is based on the premise that similar processing (in a qualitative sense) happens in human speech perception. In addition, auditory feature extraction at two scales incorporates the effects of both long term and short term distortions on speech quality. The proposed algorithm is shown to perform at least as good as the ITU-T Recommendation P.563.

The Relationship between Electrically Evoked Compound Action Potential and Speech Perception in CI24RE Implant Users

Background and Objectives : Since performance with a cochlear implant is negatively correlated with an individual's duration of deafness prior to implantation, neural survival may be an important factor in determining outcome. To date, studies relating physiological measures such as the threshold and growth of the electrically evoked potentials have not shown clear relationships to human speech perception abilities, although there are positive results correlating these measures to nerve survival in animals. The primary goal of this study was to determine the extent to which the growth functions of electrically evoked compound action potential (ECAP) are related with speech perception performance in recent CI24RE implant users and to investigate the relationship between ECAP growth function and performance according to more limited electrodes. Subjects and Method : We measured ECAP growth functions to biphasic pulses using reverse neural telemetry system in 21 CI24RE implant users. For each subject, the slope of the growth function was compared with speech perception tests. In addition, this measure was also evaluated for a more limited set of electrodes. Results : Significant correlations have been found between the slope of ECAP growth functions and speech performance in CI24RE implant recipients. In addition, a relatively strong correlation has been found between the slope of ECAP growth function and performance on word recognition test in the group of basal electrodes. Conclusion : Effectiveness of the cochlear implant can be quite variable. These results show that ECAP measure may be useful in developing a test for predicting the outcomes of using the implant.

Zebra finches exhibit speaker-independent phonetic perception of human speech

Humans readily distinguish spoken words that closely resemble each other in acoustic structure, irrespective of audible differences between individual voices or sex of the speakers. There is an ongoing debate about whether the ability to form phonetic categories that underlie such distinctions indicates the presence of uniquely evolved, speech-linked perceptual abilities, or is based on more general ones shared with other species. We demonstrate that zebra finches (Taeniopygia guttata) can discriminate and categorize monosyllabic words that differ in their vowel and transfer this categorization to the same words spoken by novel speakers independent of the sex of the voices. Our analysis indicates that the birds, like humans, use intrinsic and extrinsic speaker normalization to make the categorization. This finding shows that there is no need to invoke special mechanisms, evolved together with language, to explain this feature of speech perception.

Effects of syllable-final segment duration on the identification of synthetic speech continua by birds and humans

In an attempt to test whether experience with or knowledge of language is necessary to show typical speaking rate effects in the perception of speech, budgerigars (Melopsittacus undulatus) and humans categorized stimuli from the synthetic continua /ba/-/wa/ and /bas/-/was/, with both short and long syllable-final phonemes. This comparative approach aims to shed some light on whether knowledge of language has a role in rate normalization effects, such as using duration information as an indicator of speaking rate in human speech perception. Syllable-final phoneme durations were varied, and were either temporally adjacent to the initial target (CV series) or were nonadjacent (CVC series). The birds were always influenced by syllable-final duration variation in the present experiments and displayed greater boundary shifts than humans. In humans, there was a significant boundary shift observed in the CV series, but there were no effects of duration variation in the final segment in the CVC series. The results from the birds suggest that specialized speech-based principles may not be necessary for explaining findings of grouping speech or speechlike elements in perception.

Time-Warp–Invariant Neuronal Processing

Author SummaryThe brain has a robust ability to process sensory stimuli, even when those stimuli are warped in time. The most prominent example of such perceptual robustness occurs in speech communication. Rates of speech can be highly variable both within and across speakers, yet our perceptions of words remain stable. The neuronal mechanisms that subserve invariance to time warping without compromising our ability to discriminate between fine temporal cues have puzzled neuroscientists for several decades. Here, we describe a cellular process whereby auditory neurons recalibrate, on the fly, their perceptual clocks and allows them effectively to correct for temporal fluctuations in the rate of incoming sensory events. We demonstrate that this basic biophysical mechanism allows simple neural architectures to solve a standard benchmark speech-recognition task with near perfect performance. This proposed mechanism for time-warp–invariant neural processing leads to novel hypotheses about the origin of speech perception pathologies.

Variability of vowel productions within and between days.

Theories of speech production and speech perception assume that phonetic categories can be characterized by stable properties. For example, the notion of stable vowel targets is used to organize articulation, and such targets could serve as the acoustic basis for recognizing vowel categories. However, there is substantial variation in the acoustic patterns of phonetic categories. The lack of invariance between acoustic properties and the phonetic categories of speech has posed a theoretical problem for understanding human speech perception. Although most theories vary in how acoustic‐phonetic variability is approached, theories treat such variability as inherent in speech production or statistically regular or noise. In order to begin to understand acoustic‐phonetic variability, we examined the naturally occurring variability in speech production over time. Participants’ productions of seven different vowels ([IH], [EE], [EH], [AW], [AH], [AE], [UW]) were recorded in nine sessions: at three specific times...

Computational Auditory Scene Analysis: Principles, Algorithms and Applications

First Page