Automatic Speech Understanding Research Articles

Safety and Workload Benefits of Automatic Speech Understanding for Radar Label Updates

Air traffic controllers (ATCos) quantified the benefits of automatic speech recognition and understanding (ASRU) on workload and flight safety. As a baseline procedure, ATCos manually enter all verbal clearances into the aircraft radar labels by mouse. In our proposed solution, ATCos are supported by ASRU, which is capable of delivering the required radar label updates automatically. ATCos need to visually review the ASRU-based label updates and only have to make corrections in case of misinterpretations. Overall, the amount of time required for manually inserting clearances, i.e., by selecting the correct input in the radar labels, was reduced from 12,700 s during 14 hours of simulation time down to 405 s when ATCos were supported by ASRU. Considering the additional time of mental workload for verifying ASRU output, there is still a saving of more than one-third of the time for radar label updates. This paper also considers safety aspects, i.e., how often incorrect inputs into aircraft radar labels occur with ASRU. The number of wrong or missing inputs is less than without ASRU support. This paper advances the use case that ASRU could potentially improve safety and efficiency for ATCo operations for arrivals.

Lessons Learned in Transcribing 5000 h of Air Traffic Control Communications for Robust Automatic Speech Understanding

Voice communication between air traffic controllers (ATCos) and pilots is critical for ensuring safe and efficient air traffic control (ATC). The handling of these voice communications requires high levels of awareness from ATCos and can be tedious and error-prone. Recent attempts aim at integrating artificial intelligence (AI) into ATC communications in order to lessen ATCos’s workload. However, the development of data-driven AI systems for understanding of spoken ATC communications demands large-scale annotated datasets, which are currently lacking in the field. This paper explores the lessons learned from the ATCO2 project, which aimed to develop an unique platform to collect, preprocess, and transcribe large amounts of ATC audio data from airspace in real time. This paper reviews (i) robust automatic speech recognition (ASR), (ii) natural language processing, (iii) English language identification, and (iv) contextual ASR biasing with surveillance data. The pipeline developed during the ATCO2 project, along with the open-sourcing of its data, encourages research in the ATC field, while the full corpus can be purchased through ELDA. ATCO2 corpora is suitable for developing ASR systems when little or near to no ATC audio transcribed data are available. For instance, the proposed ASR system trained with ATCO2 reaches as low as 17.9% WER on public ATC datasets which is 6.6% absolute WER better than with “out-of-domain” but gold transcriptions. Finally, the release of 5000 h of ASR transcribed speech—covering more than 10 airports worldwide—is a step forward towards more robust automatic speech understanding systems for ATC communications.

Validating Automatic Speech Recognition and Understanding for Pre-Filling Radar Labels—Increasing Safety While Reducing Air Traffic Controllers’ Workload

Automatic speech recognition and understanding (ASRU) for air traffic control (ATC) has been investigated in different ATC environments and applications. The objective of this study was to quantify the effect of ASRU support for air traffic controllers (ATCos) radar label maintenance in terms of safety and human performance. Therefore, an implemented ASRU system was validated within a human-in-the-loop environment by ATCos in different traffic-density scenarios. In the baseline condition, ATCos performed radar label maintenance by entering verbally instructed ATC commands with a mouse and keyboard. In the proposed solution, ATCos were supported by ASRU, which achieved a command recognition rate of 92.5% with a command error rate of 2.4%. ASRU support reduced the number of wrong or missing inputs from ATCos into the radar label by a factor of two, which contemporaneously improved their situational awareness. Furthermore, ATCos where able to perform more successful secondary tasks when using ASRU support, indicating a greater capacity to handle unexpected events. The results from NASA TLX showed that the perceived workload decreased with a statistical significance of 4.3% across all scenarios. In conclusion, this study provides evidence that using ASRU for radar label maintenance can significantly reduce workload and improve flight safety.

A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers

In this paper we propose a novel virtual simulation-pilot engine for speeding up air traffic controller (ATCo) training by integrating different state-of-the-art artificial intelligence (AI)-based tools. The virtual simulation-pilot engine receives spoken communications from ATCo trainees, and it performs automatic speech recognition and understanding. Thus, it goes beyond only transcribing the communication and can also understand its meaning. The output is subsequently sent to a response generator system, which resembles the spoken read-back that pilots give to the ATCo trainees. The overall pipeline is composed of the following submodules: (i) an automatic speech recognition (ASR) system that transforms audio into a sequence of words; (ii) a high-level air traffic control (ATC)-related entity parser that understands the transcribed voice communication; and (iii) a text-to-speech submodule that generates a spoken utterance that resembles a pilot based on the situation of the dialogue. Our system employs state-of-the-art AI-based tools such as Wav2Vec 2.0, Conformer, BERT and Tacotron models. To the best of our knowledge, this is the first work fully based on open-source ATC resources and AI tools. In addition, we develop a robust and modular system with optional submodules that can enhance the system’s performance by incorporating real-time surveillance data, metadata related to exercises (such as sectors or runways), or even a deliberate read-back error to train ATCo trainees to identify them. Our ASR system can reach as low as 5.5% and 15.9% absolute word error rates (WER) on high- and low-quality ATC audio. We also demonstrate that adding surveillance data into the ASR can yield a callsign detection accuracy of more than 96%.

Acquaintance with Natural Language Processing for Building Smart Society

Natural Language Processing (NLP) deals with the spoken languages by using computer and Artificial Intelligence. As people from different regional areas using different digital platforms and expressing their views in their spoken language, it is now must to focus on working spoken languages in India to make our society smart and digital. NLP research grown tremendously in last decade which results in Siri, Google Assistant, Alexa, Cortona and many more automatic speech recognitions and understanding systems (ASR). Natural Language Processing can be understood by classifying it into Natural Language Generation and Natural Language Understanding. NLP is widely used in various domain such as Health Care, Chatbot, ASR building, HR, Sentiment analysis etc.

A new diagnostic approach for the identification of patients with neurodegenerative cognitive complaints.

Neurodegenerative diseases causing dementia are known to affect a person’s speech and language. Part of the expert assessment in memory clinics therefore routinely focuses on detecting such features. The current outpatient procedures examining patients’ verbal and interactional abilities mainly focus on verbal recall, word fluency, and comprehension. By capturing neurodegeneration-associated characteristics in a person’s voice, the incorporation of novel methods based on the automatic analysis of speech signals may give us more information about a person’s ability to interact which could contribute to the diagnostic process. In this proof-of-principle study, we demonstrate that purely acoustic features, extracted from recordings of patients’ answers to a neurologist’s questions in a specialist memory clinic can support the initial distinction between patients presenting with cognitive concerns attributable to progressive neurodegenerative disorders (ND) or Functional Memory Disorder (FMD, i.e., subjective memory concerns unassociated with objective cognitive deficits or a risk of progression). The study involved 15 FMD and 15 ND patients where a total of 51 acoustic features were extracted from the recordings. Feature selection was used to identify the most discriminating features which were then used to train five different machine learning classifiers to differentiate between the FMD/ND classes, achieving a mean classification accuracy of 96.2%. The discriminative power of purely acoustic approaches could be integrated into diagnostic pathways for patients presenting with memory concerns and are computationally less demanding than methods focusing on linguistic elements of speech and language that require automatic speech recognition and understanding.

Prosodic stress detection for fixed stress languages using formal atom decomposition and a statistical hidden Markov hybrid

The detection of prosodic events, prosodic stress, and speech segmentation based on prosody have received much attention in the research community in the past decades. Prosody is relevant for both main areas of speech technology, text-to-speech synthesis and automatic speech recognition and understanding, and is exploited increasingly: besides providing redundancy, prosody is recognized to carry information unavailable from other sources and also contributes to the naturalness of the perceived speech. This paper addresses a recently proposed intonation analysis technique, called Weighted Correlation based Atom Decomposition (WCAD). The WCAD approach is inspired by the physiology of speech production and the Fujisaki-model used in speech synthesis, however, it is employed in an analytic, and not in a generative approach: the intonation contour is decomposed into a set of elementary components, called atoms, by a pattern matching algorithm. The obtained atom decomposition is used for prosodic stress detection and automatic phonological phrasing. We compare and also combine the WCAD approach to a phonological approach, which relies on automatic segmentation for phonological phrases using a Gaussian Mixture Model (GMM) / Hidden Markov Model (HMM) model and Viterbi-alignment. Results show comparable performance of the physiologically inspired system to the phonologically conceived one in phonological phrasing for two fixed stress languages of different language families: Hungarian and French. By this we also intend to experimentally confirm that the physiologically inspired WCAD model is able to predict or extract linguistically relevant markers linked to meaning. Finally, a hybrid model is proposed, combining the physiologically and the phonologically inspired approaches, and evaluated in phonological phrase and prosodic stress detection in both languages. The performance of the hybrid model is found to be superior to both individual systems. The basic algorithmic steps targeting feature extraction and atom decomposition, as a whole, are applicable to a wide range of languages. However, linking these to linguistic levels and meaning is by nature language specific, i.e. determining which event refers to which linguistic cue or function cannot be defined without knowing the language.

Hybrid approach for conceptual segmentation of spontaneous Arabic oral utterances

This work is a part of automatic spontaneous Arabic speech understanding. In this paper, we propose a hybrid method for the conceptual segmentation of spontaneous Arabic oral utterances. Our method is composed of two main parts. The symbolic part is for the conceptual segmentation and the disfluencies processing (simple and complex). The numerical part is based on the machine learning to extract rules allowing the Out-Of-Vocabulary (OOV) words processing (unknown words and miss-recognized words). Also, we carried out the Arabic Oral Conceptual Segmentation Module (AOCSM) using the proposed method. The evaluation of AOCSM is done by evaluating each parts (symbolic and numerical) separately. The symbolic part was evaluated and compared with CSM [4]. The numerical part was evaluated through the literal understanding module of SARF system [2] using a comparative study that will involve the results generated by the two versions of the literal understanding module (i.e., literal understanding module of SARF system with and without AOCSM). The evaluation results are encouraging. The first evaluation showed an improvement in the F-measure rate of 5,12% relative to the MSC and the second evaluation showed that the acceptable understanding of the SARF system has increased by 9.31% after the integration of our AOCSM.

Developments and directions in speech recognition and understanding, Part 1 [DSP Education

To advance research, it is important to identify promising future research directions, especially those that have not been adequately pursued or funded in the past. The working group producing this article was charged to elicit from the human language technology (HLT) community a set of well-considered directions or rich areas for future research that could lead to major paradigm shifts in the field of automatic speech recognition (ASR) and understanding. ASR has been an area of great interest and activity to the signal processing and HLT communities over the past several decades. As a first step, this group reviewed major developments in the field and the circumstances that led to their success and then focused on areas it deemed especially fertile for future research. Part 1 of this article will focus on historically significant developments in the ASR area, including several major research efforts that were guided by different funding agencies, and suggest general areas in which to focus research.

Automatic Prosody Labeling Using Multiple Models for Japanese

Automatic prosody labeling is the task of automatically annotating prosodic labels such as syllable stresses or break indices into speech corpora. Prosody-labeled corpora are important for speech synthesis and automatic speech understanding. However, the subtleness of physical features makes accurate labeling difficult. Since errors in the prosodic labels can lead to incorrect prosody estimation and unnatural synthetic sound, the accuracy of the labels is a key factor for text-to-speech (TTS) systems. In particular, mora accent labels relevant to pitch are very important for Japanese, since Japanese is a pitch-accent language and Japanese people have a particularly keen sense of pitch accents. However, the determination of the mora accents of Japanese is a more difficult task than English stress detection in a way. This is because the context of words changes the mora accents within the word, which is different from English stress where the stress is normally put at the lexical primary stress of a word. In this paper, we propose a method that can accurately determine the prosodic labels of Japanese using both acoustic and linguistic models. A speaker-independent linguistic model provides mora-level knowledge about the possible correct accentuations in Japanese, and contributes to reduction of the required size of the speaker-dependent speech corpus for training the other stochastic models. Our experiments show the effectiveness of the combination of models.

Simultaneous recognition of words and prosody in the Boston University Radio Speech Corpus

This paper describes automatic speech recognition systems that satisfy two technological objectives. First, we seek to improve the automatic labeling of prosody, in order to aid future research in automatic speech understanding. Second, we seek to apply statistical speech recognition models of prosody for the purpose of reducing the word error rate of an automatic speech recognizer. The systems described in this paper are variants of a core dynamic Bayesian network model, in which the key hidden variables are the word, the prosodic tag sequence, and the prosody-dependent allophones. Statistical models of the interaction among words and prosodic tags are trained using the Boston University Radio Speech Corpus, a database annotated using the tones and break indices (ToBI) prosodic annotation system. This paper presents both theoretical and empirical results in support of the conclusion that a prosody-dependent speech recognizer—a recognizer that simultaneously computes the most-probable word labels and prosodic tags—can provide lower word recognition error rates than a standard prosody-independent speech recognizer in a multi-speaker speaker-dependent speech recognition task on radio speech.

Automatic recognition and understanding of spoken language - a first step toward natural human-machine communication

The promise of a powerful computing device to help people in productivity as well as in recreation can only be realized with proper human-machine communication. Automatic recognition and understanding of spoken language is the first step toward natural human-machine interaction. Research in this field has produced remarkable results, leading to many exciting expectations and new challenges. We summarize the development of the spoken language technology from both a vertical (chronology) and a horizontal (spectrum of technical approaches) perspective. We highlight the introduction of statistical methods in dealing with language-related problems, as this represents a paradigm shift in the research field of spoken language processing. Statistical methods are designed to allow the machine to learn structural regularities in the speech signal, directly from data, for the purpose of automatic speech recognition and understanding. Research results in spoken language processing have led to a number of successful applications, ranging from dictation software for personal computers and telephone-call processing systems for automatic call routing, to automatic sub-captioning for television broadcasts. We analyze the technical successes that support these applications. Along with an assessment of the state of the art in this broad technical field, we also discuss the limitations of the current technology, and point out the challenges that are ahead. This paper presents an accurate overview of spoken language technology as a basis to inspire future advances.

Spracherkennung — Stand der Technik, Einsatzmöglichkeiten und Perspektiven

Automatic speech recognition and understanding has been subject to intensive research for several decades. A few years ago, the first commercial applications for the technology resulting from these efforts began to emerge. It is now obvious that speech recognition technology will soon play an important role in everyday life and also in business. In this paper, we outline the basic principles of speech recognition technology and analyze its potentials and also its limitations We discuss current and possible future applications, ranging from simple command control systems and dictation systems to natural language dialogue and translation systems.

What can we learn from human processes of spoken language comprehension?

Conventional approaches for automatic speech recognition and understanding are essentially bottom-up. Namely, they are based on the assumption that speech can be decomposed reliably into the smallest units, can be recognized by identifying these units first, and then by clustering them to find larger units and their interrelationships. It is thus taken for granted that speech recognition is the prerequisite to speech understanding. The human processes of spoken language comprehension, on the contrary, are rather holistic and do not necessarily require identification of every constituent unit. This is possible since the listener has the capability of constructing the expected message while listening, and of matching it against the input message on the basis of global features, without paying much attention to the details. In the author’s view, the unit of human speech processing is considered to be not fixed, but variable in size depending on the context, situation, and knowledge. Likewise, the human processes of lexical access and parsing are considered to be quite different from those used in conventional systems for natural language processing. These issues will be discussed on the basis of studies by the author and his co-workers as well as by others.

M = Syntax + Prosody: A syntactic–prosodic labelling scheme for large spontaneous speech databases

In automatic speech understanding, division of continuous running speech into syntactic chunks is a great problem. Syntactic boundaries are often marked by prosodic means. For the training of statistical models for prosodic boundaries large databases are necessary. For the German Verbmobil (VM) project (automatic speech-to-speech translation), we developed a syntactic–prosodic labelling scheme where different types of syntactic boundaries are labelled for a large spontaneous speech corpus. This labelling scheme is presented and compared with other labelling schemes for perceptual–prosodic, syntactic, and dialogue act boundaries. Interlabeller consistencies and estimation of effort needed are discussed. We compare the results of classifiers (multi-layer perceptrons (MLPs) and n-gram language models) trained on these syntactic–prosodic boundary labels with classifiers trained on perceptual–prosodic and pure syntactic labels. The main advantage of the rough syntactic–prosodic labels presented in this paper is that large amounts of data can be labelled with relatively little effort. The classifiers trained with these labels turned out to be superior with respect to purely prosodic or syntactic labelling schemes, yielding recognition rates of up to 96% for the two-class-problem `boundary versus no boundary'. The use of boundary information leads to a marked improvement in the syntactic processing of the VM system.

Language portability of a speech understanding system

An important problem in automatic speech understanding is the transport of an existing application system to a new language. Design choices are required to keep the cost and implementation time of the porting as low as possible. One of the bottlenecks in spoken language system development is represented by data collection. When moving an application from the original language to a new one, it is very important to exploit, as much as possible, the data collected in the first language. This paper discusses the construction of a speech-based Air Travel Information Service (ATIS) for Italian, starting from (American) English. The aim of the work is to maximize the use of data available in English for the ATIS task in the construction of the Italian system. All components will be examined and proposed strategies will be evaluated by experimental tests. By just using a small Italian corpus and the ATIS data available in English, an understanding error rate of 6·7% was obtained on Italian written transcriptions. By adapting to speakers acoustic models and by training a language model on translations of transcriptions, an understanding error rate of 17·9% was obtained by considering read as well as spontaneous Italian sentences.

Interaction between prosody and discourse structure in a simulated man–machine dialogue

Automatic speech understanding systems are beginning to attain a level of sophistication where commercial applications are within reach. However, if humans and machines are ever going to communicate in a natural way, it is of vital importance that language modeling go beyond the sentence level. A profound understanding of discourse structure is required, and to this end, knowledge concerning how prosody interacts with other linguistic phenomena is needed. Not only will better prosodic modeling of discourse lead to better speech recognition/understanding, it will also yield more natural-sounding speech synthesis. This paper reports on a dialogue/prosody project at Telia Research, Sweden. A Wizard-of-Oz simulation of a computerized reservation system was used to collect realistic speech data. Fifty subjects were given three tasks each that entailed the reservation of flights, trains, car hire, and hotel reservations. To avoid linguistic influence on the subjects’ utterances, the tasks were given as maps and icons. A ToBI-style analysis was applied, adapted to meet language-specific requirements. The dialogues were analyzed with regard to phrase boundaries, tones, disfluencies, syntax (functions/categories), new versus given information, and pitch range. This paper describes our observations concerning the interaction between prosodic, syntactic, and higher-level linguistic phenomena, such as discourse structure.

Stochastic models of prosody for automatic spoken language systems

The prosody of an utterance, especially the placement and type of prominences and phrase breaks, is important in human understanding of speech. Prosody provides listeners with cues to factors such as syntactic structure, discourse information, and semantic content. Although prosody provides helpful and, sometimes, necessary cues for human listeners, it has not been used to a significant degree in automatic speech recognition or understanding systems. Some of the difficulties stem from an incomplete understanding of the relationship between prosody and higher level linguistic structures, e.g., discourse. Other barriers simply result from the intrinsically variable nature of prosodic production. In both cases, statistically modeling the relationship between prosody and the acoustic signal on one hand, and between prosody and higher level syntactic and semantic structures can be useful. The work here describes such a model, which expresses the mapping between the acoustic signal and syntactic/semantic structures, using prosody as an intermediary representation. Although the current model mainly uses syntax and word class information, extensions are described for handling some discourse phenomena. Analysis of the results using this approach in the context of the ATIS and other speech understanding tasks will be discussed. [Work supported by ARPA and NSF.]

The relationship between prosody and syntax in generation and comprehension of spoken messages

While it is generally believed that prosody of a spoken message conveys information concerning its syntactic structure, there exist many cases where prosody and syntax are discordant. The discordance, however, does not usually cause any difficulty in comprehension. The present study aims at elucidating the process whereby the prosodic structure is determined in speech production, as well as the process whereby it is utilized in speech perception. Using the contour of the voice fundamental frequency as an index, this paper shows examples of such discordance in utterances of various languages including English, Japanese, and Chinese, and presents an interpretation for the origin of the discordance, based on a model of the cognitive processes involved in message generation and speech production. The role of such discordance in speech preception and message comprehension is also discussed, referring to the cognitive processes involved. Finally, the implication of these results for speech synthesis by rule and automatic speech understanding is also mentioned.

Linking speech and language processing through prosody

In most spoken language understanding systems today, the interface between speech recognition and natural language processing is little more than a serial connection. The top scoring recognition results (a list of sentence hypotheses) are passed to the natural language system, which typically processes these taking the highest ranking recognition hypothesis that is interpretable and ignoring any acoustic information other than rank. The system can be improved by choosing the utterance interpretation based on a combination of the acoustic likelihood of a hypothesized word string and an associated language score (e.g., parse probability or semantic preference score). However, further gain is offered by incorporating a prosody score as well, since prosody can provide a link between syntactic/semantic structure and certain types of acoustic variability. In this talk, two types of computational models of prosody will be described that integrate acoustic cues with text features based on recognizer sentence hypotheses and associated parser outputs. Also, methods for using prosodic information in automatic speech understanding systems will be described. Experimental results with speech understanding in the ATIS domain will be described for three different aspects of prosody: prosodic phrasing, prominence placement, and cues to disfluencies. [Work supported by NSF and ARPA.]