Automatic Speech Processing Research Articles

Reconsidering methods for automatic speech processing: The case of the Language ENvironment Analysis (LENA) system

Starting around a decade ago, the Language Environment Analysis (LENA) system for automatic speech processing began to be used widely by both researchers and clinicians in order to derive estimates of numbers of adult words or child vocalizations during day-long audio recordings in naturalistic environments. Widespread adoption was spurred in part by a number of studies of correlational evidence of LENA’s reliability. However, confidence in reliable and valid measurement of metrics of conversational interactions entails consideration of both true accuracy (cf. false positive and false negative rates) for frame-based classification, as well as variability in error rates in counts of communicative vocalizations. This talk reviews evidence indicating highly variable error rates across naturalistic recordings for both frame-based classification and counts of communicative vocalizations. Such findings prompt reconsideration of conditions under which LENA can be used with confidence of both high reliability and high validity for measuring communicative interactions.

Extending automatic transcripts in a unified data representation towards a prosodic-based metadata annotation and evaluation

This paper describes a framework that extends automatic speech transcripts in order to accommodate relevant information coming from manual transcripts, the speech signal itself, and other resources, like lexica. The proposed framework automatically collects, relates, computes, and stores all relevant information together in a self-contained data source, making it possible to easily provide a wide range of interconnected information suitable for speech analysis, training, and evaluating a number of automatic speech processing tasks. The main goal of this framework is to integrate different linguistic and paralinguistic layers of knowledge for a more complete view of their representation and interactions in several domains and languages. The processing chain is composed of two main stages, where the first consists of integrating the relevant manual annotations in the speech recognition data, and the second consists of further enriching the previous output in order to accommodate prosodic information. The described framework has been used for the identification and analysis of structural metadata in automatic speech transcripts. Initially put to use for automatic detection of punctuation marks and for capitalization recovery from speech data, it has also been recently used for studying the characterization of disfluencies in speech. It was already applied to several domains of Portuguese corpora, and also to English and Spanish Broadcast News corpora.

Nonauditory Functions in Low-performing Adult Cochlear Implant Users.

Despite substantial benefits of cochlear implantation (CI) there is a high variability in speech recognition, the reasons for which are not fully understood. Especially the group of low-performing CI users is under-researched. Because of limited perceptual quality, top-down mechanisms play an important role in decoding the speech signal transmitted by the CI. Thereby, differences in cognitive functioning and linguistic skills may explain speech outcome in these CI subjects. Fifteen post-lingually deaf CI recipients with a maximum speech perception of 30% in the Freiburger monosyllabic test (low performer = LP) underwent visually presented neurocognitive and linguistic test batteries assessing attention, memory, inhibition, working memory, lexical access, phonological input as well as automatic naming. Nineteen high performer (HP) with a speech perception of more than 70% were included as a control. Pairwise comparison of the two extreme groups and discrimination analysis were carried out. Significant differences were found between LP and HP in phonological input lexicon and word retrieval (p = 0.0039∗∗). HP were faster in lexical access (p = 0.017∗) and distinguished more reliably between non-existing and existing words (p = 0.0021∗∗). Furthermore, HP outperformed LP in neurocognitive subtests, most prominently in attention (p = 0.003∗∗). LP and HP were primarily discriminated by linguistic performance and to a smaller extent by cognitive functioning (canonic r = 0.68, p = 0.0075). Poor rapid automatic naming of numbers helped to discriminate LP from HP CI users 91.7% of the time. Severe phonologically based deficits in fast automatic speech processing contribute significantly to distinguish LP from HP CI users. Cognitive functions might partially help to overcome these difficulties.

The state of voice user interfaces in Latin America

This paper describes and analyzes the state of research in Voice User Interfaces (VUIs) in Latin America based on the review of scientific documents published in SCOPUS from 1999 to June 2020, through a bibliometric analysis. We analyzed 419 academic papers. Although a gradual increase is observed over the years, the number of published documents has increased considerably since 2014. Brazil (44%) and Mexico (28%) are the countries with more documents published. Co-authorship occurs between Latin American countries (Brazil, Argentina, Mexico, Ecuador, and Costa Rica). However, the mayor collaboration from Latin American countries occurs with the United States, France, Germany, Spain, Portugal, the United Kingdom, and Japan. The main researched topics are studies of automatic speech recognition, artificial intelligence, speech processing, and human-computer interaction, which have grown over the past few years. Natural language processing, conversational agents, user experience, and chatbots are keywords related to more recent studies. Our analysis reveals that the primary active research developed in the short-term future are personal assistants and assistive technology using voice user interfaces.

Computer-assisted assessment of phonetic fluency in a second language: a longitudinal study of Japanese learners of French

Automatic second language (L2) speech fluency assessment has been one of the ultimate goals of several projects aiming at designing Computer-Assisted Pronunciation Training (CAPT) tools for L2 learners. Usually, three challenges must be tackled in order to solve the issues at stake: 1) Defining fluency from a threefold interdisciplinary perspective (acoustic and perceptual phonetics, computer science, L2 education); 2) Using a cost-effective algorithm; 3) Testing the procedure with actual learners’ data. Despite rapid technical developments in the field of automatic speech processing, the tools which are actually available for learners are still scarce, and most of them rely on automatic speech recognition (ASR). Moreover, most research on the topic is focusing on English as the target L2. Therefore, in this article, we address the following research questions: (a) is it possible to use a non-ASR-based low-level signal segmentation algorithm to predict human expert assessment of phonetic fluency in beginner Japanese learners of French in a text-reading task during the first stages of their learning? (b) if the answer to (a) is positive, then what are the best predictors of phonetic fluency among a set of available measures (see below for more details)? (c) is it possible to use this algorithm to monitor the evolution of phonetic fluency (and of its associated predictors) in these learners in a longitudinal study? As a first step, a corpus of French sentences read aloud by 12 Japanese learners of different proficiency levels in French was used to design a prediction system. The read-aloud speech data was perceptually annotated by three human experts on four dimensions: overall speech fluency, speech rate, regularity of speech rate, speech fluidity (i.e. smoothness of transitions between phones). Inter-rater agreement and reliability were high for all dimensions, and the average human ratings were compared with the scores provided by our prediction system. The results show strong correlations between human and automatic scores of speech rate and regularity of speech rate, and a weak correlation for speech fluidity. Automatic scores were finally combined together through a multiple linear regression model in order to predict overall speech fluency. The best model led to a correlation coefficient of .92 between automatic and human ratings, with a root-mean-square error of .38. In the second step of this study, a corpus of identical sentences read aloud four times over two years by 12 Japanese learners of French (after 4, 7, 12, and 19 months of French courses in Japan) was fed to the automatic system. The results show regular progress in overall speech fluency, which fits with the regular progress the Japanese learners under scrutiny were expected to make through their academic program in French at their university in Japan every semester. Our study suggests a positive answer to our first and third research questions, with speech rate as the best predictor to answer our second research question. In a pedagogical perspective, it seems that such a simple algorithm could be integrated in a CAPT tool to monitor learners’ progress in phonetic fluency in reading-aloud tasks.

Circumspection in using automated measures: Talker gender and addressee affect error rates for adult speech detection in the Language ENvironment Analysis (LENA) system.

Automatic speech processing devices have become popular for quantifying amounts of ambient language input to children in their home environments. We assessed error rates for language input estimates for the Language ENvironment Analysis (LENA) audio processing system, asking whether error rates differed as a function of adult talkers' gender and whether they were speaking to children or adults. Audio was sampled from within LENA recordings from 23 families with children aged 4-34 months. Human coders identified vocalizations by adults and children, counted intelligible words, and determined whether adults' speech was addressed to children or adults. LENA's classification accuracy was assessed by parceling audio into 100-ms frames and comparing, for each frame, human and LENA classifications. LENA correctly classified adult speech 67% of the time across families (average false negative rate: 33%). LENA's adult word count showed a mean +47% error relative to human counts. Classification and Adult Word Count error rates were significantly affected by talkers' gender and whether speech was addressed to a child or an adult. The largest systematic errors occurred when adult females addressed children. Results show LENA's classifications and Adult Word Count entailed random - and sometimes large - errors across recordings, as well as systematic errors as a function of talker gender and addressee. Due to systematic and sometimes high error in estimates of amount of adult language input, relying on this metric alone may lead to invalid clinical and/or research conclusions. Further validation studies and circumspect usage of LENA are warranted.

Assessing the accuracy of existing forced alignment software on varieties of British English

AbstractThis paper presents an analysis of the performance and usability of automatic speech processing tools on six different varieties of English spoken in the British Isles. The tools used in the present study were developed for use with Mainstream American English, but we demonstrate that their forced alignment functionality nonetheless performs extremely well on a range of British varieties, encompassing both careful and casual speech. Where phone boundary placement is concerned, substantial errors in alignment occur infrequently, and although small displacements between aligner-placed and human-placed phone boundaries are found regularly, these will rarely have a significant effect on measurements of interest for the researcher. We demonstrate that gross phone boundary placement errors, when they do arise, are particularly likely to be introduced in fast speech or with varieties that are radically different from Mainstream American English (e.g. Scots). We also observe occasional problems with phonetic transcription. Overall, we advise that, although forced alignment software is highly reliable and improving continuously, human confirmation is needed to correct errors which can displace entire stretches of speech. Nevertheless, sociolinguists can be assured that the output of these tools is generally highly accurate for a wide range of varieties.

Minimum of Information Divergence Criterion for Signals with Tuning to Speaker Voice in Automatic Speech Recognition

It is considered a problem of automatic speech recognition at basic, phonetic level of speech signal processing. It is researched a problem of noise-immunity increase. For its solution it is proposed a criterion of minimum information divergence of the signals with tuning to a speaker voice and automatic scaling of speech template to thin structure of observed (current) speech frame. An example of its practical realization is considered, efficiency characteristics are researched. Applying the author’s software we carry out an experiment and obtain qualitative estimations of threshold signals gain in case of application of proposed criterion. It is shown than this gain can be 10 dB and greater under certain conditions. Obtained results and drawn conclusions are intended it to their application for development and modernization of existent systems and techniques of automatic processing and recognition of speech intended it to operation in conditions of intensive noise effect.

Cross-Platform Mobile CALL Environment for Pronunciation Teaching and Learning

Mobile technologies promote computer-assisted language learning (CALL) while mobile applications, being learner-oriented by design, provide a powerful founding to build individual self-paced environments for language study. Mobile CALL (MALL) tools are able to offer new educational contexts and fix, at least, partially, the problems of previous generations of CALL software. Nonetheless, mobile technologies alone are not able to respond to CALL challenges without cooperation and interaction with language theory and pedagogy. To facilitate and formalize this interaction several criteria sets for CALL software has been worked out in recent years. That is why an approach based on using mobile devices is a natural way to transfer the learning process from teaching-centered classroom to a process, which is oriented to individual learners and groups of learners with better emphasis on supporting individual learning styles, user collaboration and different teaching strategies. Pronunciation teaching technology in one of areas, where the automated speech processing algorithms and corresponding software meet the problems of practical phonology. Computer-assisted prosody teaching (CAPT), a sub-domain of CALL, is a relatively new topic of interest for computer scientists and software developers. Present-day advancement of mobile CAPT tools is supported by evolutionary processes in the theory of language learning and teaching. This paper explores language–technology relations using a case of StudyIntonation – a cross-platform multi-functional mobile CAPT tool based on a digital processing core for speech processing, visualization and estimation developed by the authors. We particularly address the problems of developing CAPT evaluation frameworks. To define the problematic points of the project and understand the directions for future work, we discuss an approach to formalized evaluation using a set of CAPT-specific criteria drawing attention to such evaluation factors as general descriptive information, instructional purposes, functionality, usability, and presentation.

Method for Measuring the Intelligibility of Speech Signals in the Kullback–Leibler Information Metric

We consider the problem of determination of the intelligibility of speech of a speaker according to a finite fragment of the speech signal. It is shown that the main difficulties in the solution of this problem are connected with the necessity of analysis of small samples. To overcome the problem of small samples, we proposed a new high-speed method for measuring the intelligibility of speech signals on the sonic level of its perception. The proposed method is based on the information indicator of speech intelligibility in the Kullback-Leibler metric. We consider an example of practical realization of the new method with the use of a self-regression model of minimum sound units from the speech flow of a speaker. The characteristics of efficiency of the new method are analyzed. It is shown that, under certain conditions, the application of the information indicator enables us to realize the general systems principle of guaranteed result. On the basis of the software developed by the authors, we designed and performed full-scale experiments and established quantitative estimates for the speed of this method. It is shown that, with the help of this method, quite accurate and reliable estimates of the information indicator are obtained for short (2–3 min) segments of speech signals. The accumulated results and the conclusions made on their basis are intended for applications in the development of new systems and improvement of the existing systems of automatic speech processing and recognition intended for the operation in the real-time mode.

Calculation of Phonetic Distances between Speech Sounds

ABSTRACT A new formalism to numerically measure phonetic differences between speech sounds treating feature values of the compared phones as independent parameters that give rise to corresponding Euclidean distances is put forward. The articulatory and acoustic methods within this formalism were compared, where the corresponding results display good agreement. The more reliable and more universal character of the acoustic approach is provided by robust and precise acoustic parameters used therein. The theoretical model and the findings of this article comply also with the experimental phonetic results. The proposed approach contributes to formalization of the procedure of phone comparison and mapping needed for automatic text and speech processing.

Perplexity – a new predictor of cognitive changes in spoken language? – results of the Interdisciplinary Longitudinal Study on Adult Development and Aging (ILSE)

Abstract In addition to memory loss, progressive deterioration of speech and language skills is among the main symptoms at the onset of Alzheimer’s disease (AD) as well as in mild cognitive impairment (MCI). Detailed interview analyses demonstrated early symptoms years before the onset of AD/MCI. Automatic speech processing could be a promising approach to identifying underlying mechanisms in larger studies or even support diagnostics. Perplexity as a measure of predictability of text could be a sensitive indicator of cognitive deterioration. Therefore, voice recordings from the Interdisciplinary Longitudinal Study on Adult Development and Aging were analyzed with regard to neuropsychological parameters in participants that develop MCI/AD or remain cognitively healthy. Preliminary results indicate that perplexity predicts severity of cognitive deficits and information processing speed obtained 10–12 years later in participants who developed MCI/AD in contrast to those who stayed healthy. Findings support the heuristic value of research on the diagnostic potential of automatic speech processing.

Preschoolers initiate more conversations than their parents

Children develop, learn, and refine the complex rules of how to have a conversation beginning in their preschool years. Recent work on conversational exchanges and language usage within families has shown differences (and in some cases, similarities) in how mothers, fathers, girls, and boys interact and converse with each other. It has been suggested that mothers contribute more child-oriented and child-driven exchanges, fathers contribute more formal language that includes problem-solving and linguistic manipulation, girls seek to maintain relationships, and boys seek to establish dominance and attract or maintain and audience. These factors may influence the roles each interlocutor plays in a communicative exchange. One aspect of verbal interaction is who initiates a conversational exchange. This study examined 134 daylong audio recordings using automated speech processing techniques to estimate the frequency of conversation initiation for mothers, fathers, girls, and boys. We found that children initiate conversations most frequently, followed by mothers, followed by fathers. We found no rate difference between girls and boys. Results are consistent with the Bridge Hypothesis or Apprenticeship Model in which interlocutors are motivated in part by their social role, in this case by the social modeling parents demonstrate for their children.

Daylong acoustic recordings of family and child speech using the HomeBank database

HomeBank (https://homebank.talkbank.org/) is an online database of multi-hour, naturalistic audio recordings of child and family everyday experiences. Corpora in the database include (1) raw audio recordings; (2) corpus metadata including for example social details, standardized test scores, disability reports and status, family data such as number and quantity of siblings, orthographic transcriptions, output of diarization or automatic-speech recognition processing; and (3) tools for analyzing the data (https://github.com/homebankcode/) in a variety of domains. There are currently over a dozen corpora representing over 1100 multi-hour (often daylong) audio recordings. Use of the database is increasing in the scientific community, including researchers of speech, language, computer science, digital signal processing, automatic speech recognition, health sciences, and human development. The utility and extensibility of HomeBank is demonstrated in this talk with several current and ongoing projects that make critical use of the data. We discuss diarization and automatic speech processing techniques, speech and language use in pre-industrial families and groups, early perception and processing in infants, family speech and language dynamics in families of children with disorders, and database management. We are actively soliciting both users and contributors to the database.

Longitudinal development of conversational exchanges in children with hearing loss

Conversations between adults and children have been shown to be an important factor affecting the development of cognition, academic performance, and language use in children. Conversational exchange frequency between parents and children has been shown to increase with age. It has also been demonstrated that degraded sensory input, such as with hearing loss, can result in reduced language, speech, and communication skills in children. This study examines the longitudinal development of conversational exchanges between children with hearing loss and their parents. Thirty-nine families contributed 370 daylong recordings constituting 4337 h of audio collected from a body-worn audio recorder. Automatic speech processing techniques were used to identify and tally conversational exchanges. The main longitudinal effect of age on conversation exchange rate is confirmed, but the effect is weaker and may have a different developmental trajectory for families of children with a hearing loss. This result may be influenced by differences in the development of joint attention between children who are typically developing and those with hearing loss.

Syllable production in hard-of-hearing preschoolers

Expressive language in preschoolers has been shown to be positively related to later language development, although not all studies have shown robust effects. Most studies that consider sex have shown that girls develop expressive language earlier than boys. It has also been widely demonstrated that children with a variety of speech-, language-, and hearing-related disorders show deficits in expressive language skills. Children with hearing loss in particular have shown deficits in consonant production, prosodic control, vowel duration, and mean length of utterance. This study examines the expressive language of preschool children with hearing loss using estimates of syllable production from 366 daylong audio recordings totaling over 4348 h of audio processed with unsupervised automatic speech processing techniques. We found no main effect of sex, but typically-developing children were more voluble on average. Unexpectedly, an interaction effect suggests that typically developing boys may be driving the observed differences. This work adds to the evidence of social and biological variability in the speech production of children, and is further proof-of-concept of developmental speech production work using massive data sets and automatic analysis methods.

Understanding acoustic-phonetic environments of prelingual children with cochlear implants: Challenges, tools, and insights

Variability in auditory and linguistic environments experienced by prelingual children with cochlear implants (CIs) potentially helps explain variation in their language outcomes. Here, results are presented from several studies from our lab investigating early acoustic-phonetic environments of children with CIs. One collaborative study conducted over 10 years investigated individual differences in acoustic-phonetic quality of maternal speech as predictors of language outcomes in children with CIs. Results showed that properties of maternal speech recorded in the lab—as indexed in part by measures derived from vowel formants and fundamental frequency—significantly predicted infants’ language outcomes and growth on multiple standardized assessments two years after cochlear implantation. Other work has examined the usefulness of a widely adopted commercial automatic speech processing technology—the Language ENvironment Analysis (LENA) system—for investigating individual differences in auditory environments. Results from our study suggested considerable variability across samples in LENA’s accuracy at identifying adult speech, limiting LENA’s value for investigating individual differences in acoustic-phonetic input to children with CIs. Finally, an update is provided on current work using combined signal processing and hand-coding approaches aimed at investigating variability in acoustic input to children with CIs in their home environments. [Work supported by NIH grant R01DC008581.]

Spoken grammar practice in CALL: The effect of corrective feedback and education level in adult L2 learning

Although most second language learning studies involve high-educated participants, addressing low-educated participants seems important as research suggests that education background impacts second language (L2) learning efficiency. A controversial point in the L2 learning literature is the role of corrective feedback (CF). Research on CF in relation to individual differences, including education background, requires controlled experimental environments. This article presents a study in such a controlled environment: a computer assisted language learning (CALL) system with logging capabilities and automatic speech processing to provide CF on spoken grammar practice. In the experiment, 68 Dutch L2 learners were randomly assigned to either an automatic grammaticality CF condition or a NOCF (no corrective feedback) condition. We analysed learning outcome, evaluative measures, and practice behaviour logged by the CALL system. Practice behaviour differed with education background, which in turn related to learning outcome. Learners with high and medium education levels benefitted from speaking practice in both conditions, while low-educated learners did not benefit from practice, in neither condition. For medium and high educated learners there was a trend for CF to increase practice effectiveness. Education level therefore is an important individual difference in L2 learning research that influences the effectiveness of CF.

Robust speech processing in human auditory cortex

The brain empowers humans with remarkable abilities to navigate their acoustic environment in highly degraded conditions. This seemingly trivial task for normal hearing listeners is extremely challenging for individuals with auditory pathway disorders, and has proven very difficult to model and implement algorithmically in machines. In this talk, I will present the result of an interdisciplinary research effort where invasive and non-invasive neural recordings from human auditory cortex are used to determine the representational and computational properties of robust speech processing in the human brain. These findings show that speech processing in the auditory cortex is dynamic and adaptive. These intrinsic properties allow a listener to filter out irrelevant sound sources, resulting in a reliable and robust means of communication. Furthermore, incorporating the functional properties of neural mechanisms in speech processing models greatly impact the current models of speech perception and at the same time, lead to human-like automatic speech processing technologies.

Automatic Detection Technique for Speech Recognition based on Neural Networks Inter-Disciplinary

Automatic speech recognition allows the machine to understand and process information provided orally by a human user. It consists of using matching techniques to compare a sound wave to a set of samples, usually composed of words but also of phonemes. This field uses the knowledge of several sciences: anatomy, phonetics, signal processing, linguistics, computer science, artificial intelligence and statistics. The latest acoustic modeling methods provide deep neural networks for speech recognition. In particular, recurrent neural networks (RNNs) have several characteristics that make them a model of choice for automatic speech processing. They can keep and take into account in their decisions past and future contextual information. This paper specifically studies the behavior of Long Short-Term Memory (LSTM)-based neural networks on a specific task of automatic speech processing: speech detection. LSTM model were compared to two neural models: Multi-Layer Perceptron (MLP) and Elman’s Recurrent Neural Network (RNN). Tests on five speech detection tasks show the efficiency of the Long Short-Term Memory (LSTM) model.