Spontaneous Speech Recognition Research Articles

Аналитический обзор методов автоматического анализа экстралингвистических компонентов спонтанной речи

The accuracy of automatic spontaneous speech recognition systems is far from that of trained speech recognition systems. This is due to the fact that spontaneous speech is not as smooth and failure-free as spontaneous speech. Spontaneous speech varies from speaker to speaker: the quality of phonemes’ pronunciation, the presence of pauses, speech disruptions and extralinguistic items (laughing, coughing, sneezing, and chuckling when expressing emotions of irritation, etc.) interrupt the fluency of verbal speech. However, it is worth noting that extralinguistic items very often carry important paralinguistic information, so it is crucial for automatic spontaneous speech recognition systems not only to identify such phenomena and distinguish them from the verbal components of speech but also to classify them. This review presents an analysis of works on the topic of automatic detection and analysis of extralinguistic items in spontaneous speech. Both individual methods and approaches to the recognition of extralinguistic items in a speech stream, and works related to the multiclass classification of isolatedly recorded extralinguistic units are considered and described. The most popular methods of extralinguistic units’ analysis are neural networks, such as deep neural networks and networks based on transformer models. The basic concepts related to the term extralinguistic items are given, the original systematization of extralinguistic items in the Russian language is proposed, the corpus and databases of audio spoken speech both in Russian and in other languages are described, the data sets of extralinguistic items recorded isolatedly are also given. The accuracy of extralinguistic items recognition increases with the following conditions of work with the speech signal: pre-processing of audio signals of items has shown an increase in the accuracy of separately recorded extralinguistic items classification; consideration of context (analysis of several frames of speech signal) and use of filters for smoothing the time series after extraction of feature vectors showed an increase in accuracy in frame-by-frame analysis of the speech signal with spontaneous speech.

Using Spontaneous Speech Recognition as a Biomarker to Distinguish Dementia Patients

AbstractBackgroundDigital biomarkers such as individual speech recordings, are a cost‐effective tool for assessing cognitive status using linguistic and acoustic content. The purpose of this study is to investigate whether Natural Language Processing (NLP) methods and audio feature extraction can distinguish spontaneous speech outputs in dementia patients from a healthy population.Method240 audio samples and transcripts of clinician‐participant dialogues involving the Cookie‐Theft visual description were acquired from the DementiaBank Pitt corpus (Dementia group (DG) n = 169; healthy controls (CG) n = 71). Lexical variety, syntactic complexity, salience of information, disfluency patterns, and sentiment analysis were evaluated. Sound synthesis and an acoustic R analysis package were used to filter and examine acoustic properties of recorded audio such as silence and speech segmentation, pitch tracking, and frequency modulation of vibrato or jitter. DG and CG were compared using non‐parametric Kruskal‐Wallis tests, Dunn’s multiple comparison, and Mann‐Whitney U‐test in the statistical analysis. In comparison to CG, we hypothesized that DG would differ in levels of lexical diversity, grammatical complexity, and overall salience.ResultOverall, text analysis showed no variations in disfluencies between groups, but there were significant differences in lexical diversity, syntactic complexity and reference rate to reality (p<0.0001). Additionally, there were no appreciable differences in the frequency of salient information between the two groups, however the frequency of non‐salient information and disfluencies was considerably higher in the DG (p<0.05). Analysis of acoustic features revealed that DG had longer average speech duration and speech segment durations compared to CG (p<0.001).. On the other hand, the rate of change of acoustic characteristics including pitch, mean entropy, and harmonics‐to‐noise ratios did not significantly differ between DG and CG.ConclusionIn conclusion, this paradigm reveals that, in comparison to healthy controls, dementia patients exhibited increased lexical diversity yet lower linguistic complexity, and a shift towards characterizing non‐salient objects in given stimuli, as well as longer segments of speech. Further examination of the rhythmic and acoustic characteristics of speech may provide a comprehensive framework towards defining speech as a potential digital biomarker for dementia risk, and provide applications to regions experiencing increasing socioeconomic disparities.

Few-Shot Learning in Emotion Recognition of Spontaneous Speech Using a Siamese Neural Network With Adaptive Sample Pair Formation

Speech-based machine learning (ML) has been heralded as a promising solution for tracking prosodic and spectrotemporal patterns in real-life that are indicative of emotional changes, providing a valuable window into one's cognitive and mental state. Yet, the scarcity of labelled data in ambulatory studies prevents the reliable training of ML models, which usually rely on "data-hungry" distribution-based learning. Leveraging the abundance of labelled speech data from acted emotions, this paper proposes a few-shot learning approach for automatically recognizing emotion in spontaneous speech from a small number of labelled samples. Few-shot learning is implemented via a metric learning approach through a siamese neural network, which models the relative distance between samples rather than relying on learning absolute patterns of the corresponding distributions of each emotion. Results indicate the feasibility of the proposed metric learning in recognizing emotions from spontaneous speech in four datasets, even with a small amount of labelled samples. They further demonstrate superior performance of the proposed metric learning compared to commonly used adaptation methods, including network fine-tuning and adversarial learning. Findings from this work provide a foundation for the ambulatory tracking of human emotion in spontaneous speech contributing to the real-life assessment of mental health degradation.

Lahjoita puhetta: a large-scale corpus of spoken Finnish with some benchmarks

The Donate Speech campaign has so far succeeded in gathering approximately 3600 h of ordinary, colloquial Finnish speech into the Lahjoita puhetta (Donate Speech) corpus. The corpus includes over twenty thousand speakers from all the regions of Finland and from all age brackets. The primary goals of the collection were to create a representative, large-scale resource to study spontaneous spoken Finnish and to accelerate the development of language technology and speech-based services. In this paper, we present the collection process and the collected corpus, and showcase its versatility through multiple use cases. The evaluated use cases include: automatic speech recognition of spontaneous speech, detection of age, gender, dialect and topic and metadata analysis. We provide benchmarks for the use cases, as well downloadable, trained baseline systems with open-source code for reproducibility. One further use case is to verify the metadata and transcripts given in this corpus itself, and to suggest artificial metadata and transcripts for the part of the corpus where it is missing.

Investigation of the Effect of Increased Dimension Levels in Speech Emotion Recognition

In human-machine interaction systems, speech emotion recognition plays a key role. Recognition of categorical emotions has made a great improvement during the last few decades, but emotion recognition of spontaneous speech is still very challenging. This paper aims to investigate emotion recognition from the spontaneous speech in the three-dimensional model. Each dimension represents one primitive, generic attribute of an emotion. Middle levels of each dimension were introduced in this paper. LSTM network was employed to estimate the dimensions due to its effectiveness in speech emotion recognition. In the experiments, we use the IEMOCAP database and the accuracy is 30-35%. The confusion matrixes show that our method leads to a more concentrated dimension location. Furthermore, dimensions were applied in categorical emotion recognition. This indicates that increasing dimension levels could provide a possibility of dimension estimation, and suggests that it is possible to promote speech emotion recognition with dimensions.

Semi-Natural and Spontaneous Speech Recognition Using Deep Neural Networks with Hybrid Features Unification

Recently, identifying speech emotions in a spontaneous database has been a complex and demanding study area. This research presents an entirely new approach for recognizing semi-natural and spontaneous speech emotions with multiple feature fusion and deep neural networks (DNN). A proposed framework extracts the most discriminative features from hybrid acoustic feature sets. However, these feature sets may contain duplicate and irrelevant information, leading to inadequate emotional identification. Therefore, an support vector machine (SVM) algorithm is utilized to identify the most discriminative audio feature map after obtaining the relevant features learned by the fusion approach. We investigated our approach utilizing the eNTERFACE05 and BAUM-1s benchmark databases and observed a significant identification accuracy of 76% for a speaker-independent experiment with SVM and 59% accuracy with, respectively. Furthermore, experiments on the eNTERFACE05 and BAUM-1s dataset indicate that the suggested framework outperformed current state-of-the-art techniques on the semi-natural and spontaneous datasets.

KsponSpeech: Korean Spontaneous Speech Corpus for Automatic Speech Recognition

This paper introduces a large-scale spontaneous speech corpus of Korean, named KsponSpeech. This corpus contains 969 h of general open-domain dialog utterances, spoken by about 2000 native Korean speakers in a clean environment. All data were constructed by recording the dialogue of two people freely conversing on a variety of topics and manually transcribing the utterances. The transcription provides a dual transcription consisting of orthography and pronunciation, and disfluency tags for spontaneity of speech, such as filler words, repeated words, and word fragments. This paper also presents the baseline performance of an end-to-end speech recognition model trained with KsponSpeech. In addition, we investigated the performance of standard end-to-end architectures and the number of sub-word units suitable for Korean. We investigated issues that should be considered in spontaneous speech recognition in Korean. KsponSpeech is publicly available on an open data hub site of the Korea government.

AutoSSR: an efficient approach for automatic spontaneous speech recognition model for the Punjabi Language

In this article, the authors have presented the design and development of automatic spontaneous speech recognition of the Punjabi language. To dimensions up to the natural speech recognizer, the very large vocabulary Punjabi text corpus has been taken from a Punjabi interview’s speech corpus, presentations, etc. Afterward, the Punjabi text corpus has been cleaned by using the proposed corpus optimization algorithm. The proposed automatic spontaneous speech model has been trained with 13,218 of Punjabi words and more than 200 min of recorded speech. The research work also confirmed that the 2,073,456 unique in-word Punjabi tri-phoneme combinations present in the dictionary comprise of 131 phonemes. The performance of the proposed model has grown increasingly to 87.10% sentence-level accuracy for 2381 Punjabi trained sentences and word-level accuracy of 94.19% for 13,218 Punjabi words. Simultaneously, the word error rate has been reduced to 5.8% for 13,218 Punjabi words. The performance of the proposed system has also been tested by using other parameters such as overall likelihood per frame and convergence ratio on various iterations for different Gaussian mixtures.

Data augmentation on spontaneous Indonesian automatic speech recognition using statistical machine translation

Language model plays an important role in decoding process of the automatic speech recognition. The accuracy of spontaneous speech recognition is still very low compared to dictated speech of the Indonesian automatic speech recognition. It is due to the lack of the number of spontaneous data. Collecting spontaneous data is also difficult to do, so one of the candidate solutions is to augment data from existing spontaneous data. In this research, experiments are conducted on language models to improve the accuracy of spontaneous Indonesian speech recognition by conducting data augmentation. Data augmentation in this research is done by using statistical machine translation named ‘Moses’. Language modeling technique used here is n-gram. GMM-HMM is used for acoustic modeling. First, spontaneous text corpus is added to the text corpus, then the data augmentation is conducted. When the language model is formed from the addition of a spontaneous text corpus, there is an increase in accuracy of 3.59% relative to the baseline. When data augmentation is done on language model there is an increase in accuracy of 2.74% relative to the baseline. However, this decrease is considered not significant compared to the effort required in collecting spontaneous data manually.

Acoustic Data-Driven Subword Units Obtained through Segment Embedding and Clustering for Spontaneous Speech Recognition

We propose a method to extend a phoneme set by using a large amount of broadcast data to improve the performance of Korean spontaneous speech recognition. In the proposed method, we first extract variable-length phoneme-level segments from broadcast data and then convert them into fixed-length embedding vectors based on a long short-term memory architecture. We use decision tree-based clustering to find acoustically similar embedding vectors and then build new acoustic subword units by gathering the clustered vectors. To update the lexicon of a speech recognizer, we build a lookup table between the tri-phone units and the units derived from the decision tree. Finally, the proposed lexicon is obtained by updating the original phoneme-based lexicon by referencing the lookup table. To verify the performance of the proposed unit, we compare the proposed unit with the previous units obtained by using the segment-based k-means clustering method or the frame-based decision-tree clustering method. As a result, the proposed unit is shown to produce better performance than the previous units in both spontaneous, and read Korean speech recognition tasks.

Identification of Spontaneous Spoken Texts in Slovak

Abstract We propose a text classification method for the purpose of creating a language model for automatic recognition of spontaneous spoken speech. Transcripts from our departmental speech database served as spontaneous spoken texts. Using supervised machine learning methods, we have created multiple classification models (including neural networks), that were able to distinguish them from written texts with high accuracy. We subsequently verified the accuracy of our trained models on a database of texts containing direct speech extracted from newspaper articles.

Automatic speech recognizers for Mexican Spanish and its open resources

Development of automatic speech recognition systems relies on the availability of distinct language resources such as speech recordings, pronunciation dictionaries, and language models. These resources are scarce for the Mexican Spanish dialect. In this work, we present a revision of the CIEMPIESS corpus that is a resource for spontaneous speech recognition in Mexican Spanish of Central Mexico. It consists of 17 h of segmented and transcribed recordings, a phonetic dictionary composed by 53,169 unique words, and a language model composed by 1,505,491 words extracted from 2489 university newsletters. We also evaluate the CIEMPIESS corpus using three well known state of the art speech recognition engines, having satisfactory results. These resources are open for research and development in the field. Additionally, we present the methodology and the tools used to facilitate the creation of these resources which can be easily adapted to other variants of Spanish, or even other languages.

An automatic speech recognition system for spontaneous Punjabi speech corpus

Automatic speech recognition is the central part of the wheel towards the natural person-to-machine interaction technique. Due to the high disparity of speaking styles, speech recognition surely demands composite methods to constitute this irregularity. A speech recognition method can work in numerous distinct states such as speaker dependent/independent speech, isolated/continuous/spontaneous speech recognition, for less to very large vocabulary. The Punjabi language is being spoken by concerning 104 million peoples in India, Pakistan and other countries with Punjabi migrants. The Punjabi language is written in Gurmukhi writing in Indian Punjab, while in Shahmukhi writing in Pakistani Punjab. In the paper, the objective is to build the speaker independent automatic spontaneous speech recognition system for the Punjabi language. The system is also capable to recognize the spontaneous Punjabi live speech. So far, no work has to be achieved in the area of spontaneous speech recognition system for the Punjabi language. The user interfaces for Punjabi live speech system is created by using the java programming. Till now, automatic speech system is trained with 6012 Punjabi words and 1433 Punjabi sentences. The performance measured in terms of recognition accuracy which is 93.79% for Punjabi words and 90.8% for Punjabi sentences.

Croatian Large Vocabulary Automatic Speech Recognition

This paper presents procedures used for development of a Croatian large vocabulary automatic speech recognition system (LVASR). The proposed acoustic model is based on context-dependent triphone hidden Markov models and Croatian phonetic rules. Different acoustic and language models, developed using a large collection of Croatian speech, are discussed and compared. The paper proposes the best feature vectors and acoustic modeling procedures using which lowest word error rates for Croatian speech are achieved. In addition, Croatian language modeling procedures are evaluated and adopted for speaker independent spontaneous speech recognition. Presented experiments and results show that the proposed approach for automatic speech recognition using context-dependent acoustic modeling based on Croatian phonetic rules and a parameter tying procedure can be used for efficient Croatian large vocabulary speech recognition with word error rates below 5%.

Automatic Spontaneous Speech Recognition for Punjabi Language Interview Speech Corpus

Automatic Speech Recognition presents natural phenomena for the communication among man and machine. The purpose of Speech Recognition speech system is to convert the sequence of sound units in the form of text description. The main objective of the research work is to develop the automatic spontaneous speech model for the Punjabi language. Punjabi is categorized as a constituent of the Indo-Aryan subgroup of the Indo-European family of languages. So far no work has been done in the field of spontaneous Punjabi speech recognition system. In spontaneous speech system, the sounds are usually unprompted and non- designed and are commonly described by repetitions, repairs, false start, partial words and non-planned words, silence gap etc. In this paper, the focus is on the development of the spontaneous speech model for the recognition of the Punjabi language. The GUI for Punjabi speech model also has been created and tested for the live Punjabi interview speech corpus. The recognition accuracy is 98.6% for Punjabi sentences and 98.8% for Punjabi words. The sphinx toolkit and java programming are used to build a spontaneous speech model for Punjabi live speech.

Two-step algorithm of training initialization for acoustic models based on deep neural networks

This paper presents a two-step initialization algorithm for training of acoustic models based on deep neural networks. The algorithm is focused on reducing the impact of the non-speech segments on the acoustic model training. The idea of the proposed algorithm is to reduce the percentage of non-speech examples in the training set. Effectiveness evaluation of the algorithm has been carried out on the example of English spontaneous telephone speech recognition (Switchboard). The application of the proposed algorithm has led to 3% relative word error rate reduction, compared with the training initialization by restricted Boltzmann machines. The results presented in the paper can be applied in the development of automatic speech recognition systems.

Speaker-dependent features for spontaneous speech recognition

Speaker-dependent features for spontaneous speech recognition

The recognition of read and spontaneous speech in local vernacular: The case of Zurich German

Listeners are typically able to identify speech as either being produced spontaneously or read from a transcript. In the present research we investigated whether this is true in vernacular speech when typical cues to read and spontaneous speech are either missing and/or ambiguous. In addition it was investigated what the acoustic cues for listeners' identification ability were. 26 listeners of Zurich German participated in two perception experiments. In Experiment I, listeners judged 128 stimuli (64 spontaneous, 64 read) in a two-alternative identification task as either spontaneous or read. Results revealed that overall listener performance was well above chance (mean A′=0.82) while there was no bias for either read or spontaneous speech (mean BD″=0). There were significant effects of speaker and listener in A′ and BD″. In Experiment II, the same 26 listeners rated the same 64 read speech stimuli from Experiment I as to whether they sounded more or less read. Results revealed that there was considerable within-category variability as a function of speaker. From eight acoustic prosodic parameters only articulation rate explained listener behavior to some degree in Experiments I and II. Overall the study suggested that read and spontaneous speech can be recognized based on very subtle cues to these speaking styles.

Less than careful speech: Exploring the roles of target duration and time varying intensity in spoken language processing

Despite substantial phonetic variation in the production of natural speech, listeners are quite adept at processing varying speech signals. Further, the recognition of spontaneous speech is a multifaceted problem where listeners are drawing information from many sources in the recognition process. Previous research has begun to outline the influence of both syntactic and semantic contributions to spoken word recognition; however, it has been argued (Ernestus et al., 2002; van de Ven et al., 2012) that such cues reach a ceiling in their contributions, and that acoustic information also aids the listener when processing language. There has been limited discussion on what information encoded within a given speech signal lends itself to more effectively processing spontaneous speech. The present study contributes to such a discussion, using responses from 85 participants in modified Cloze tasks and speech in noise manipulations, where results serve as a starting place to describe the contributions of duration and intensity in natural speech (as well as their relative effects on spoken language processing). The introduction of particular acoustic information—over and above syntactic and semantic cues—has been observed within this study as a statistically significant contributor to more effective spoken language processing.

Probabilistic speech feature extraction with context-sensitive Bottleneck neural networks

We introduce a novel context-sensitive feature extraction approach for spontaneous speech recognition. As bidirectional Long Short-Term Memory (BLSTM) networks are known to enable improved phoneme recognition accuracies by incorporating long-range contextual information into speech decoding, we integrate the BLSTM principle into a Tandem front-end for probabilistic feature extraction. Unlike the previously proposed approaches which exploit BLSTM modeling by generating a discrete phoneme prediction feature, our feature extractor merges continuous high-level probabilistic BLSTM features with low-level features. By combining BLSTM modeling and Bottleneck (BN) feature generation, we propose a novel front-end that allows us to produce context-sensitive probabilistic feature vectors of arbitrary size, independent of the network training targets. Evaluations on challenging spontaneous, conversational speech recognition tasks show that this concept prevails over recently published architectures for feature-level context modeling.