Speech Understanding System Research Articles

Lessons Learned in Transcribing 5000 h of Air Traffic Control Communications for Robust Automatic Speech Understanding

Voice communication between air traffic controllers (ATCos) and pilots is critical for ensuring safe and efficient air traffic control (ATC). The handling of these voice communications requires high levels of awareness from ATCos and can be tedious and error-prone. Recent attempts aim at integrating artificial intelligence (AI) into ATC communications in order to lessen ATCos’s workload. However, the development of data-driven AI systems for understanding of spoken ATC communications demands large-scale annotated datasets, which are currently lacking in the field. This paper explores the lessons learned from the ATCO2 project, which aimed to develop an unique platform to collect, preprocess, and transcribe large amounts of ATC audio data from airspace in real time. This paper reviews (i) robust automatic speech recognition (ASR), (ii) natural language processing, (iii) English language identification, and (iv) contextual ASR biasing with surveillance data. The pipeline developed during the ATCO2 project, along with the open-sourcing of its data, encourages research in the ATC field, while the full corpus can be purchased through ELDA. ATCO2 corpora is suitable for developing ASR systems when little or near to no ATC audio transcribed data are available. For instance, the proposed ASR system trained with ATCO2 reaches as low as 17.9% WER on public ATC datasets which is 6.6% absolute WER better than with “out-of-domain” but gold transcriptions. Finally, the release of 5000 h of ASR transcribed speech—covering more than 10 airports worldwide—is a step forward towards more robust automatic speech understanding systems for ATC communications.

Adversarial Training for Multi Domain Dialog System

Natural Language Understanding and Speech Understanding systems are now a global trend, and with the advancement of artificial intelligence and machine learning techniques, have drawn attention from both the academic and business communities. Domain prediction, intent detection and entity extraction or slot fillings are the most important parts for such intelligent systems. Various traditional machine learning algorithms such as Bayesian algorithm, Support Vector Machine, and Artificial Neural Network, along with recent Deep Neural Network techniques, are used to predict domain, intent, and entity. Most language understanding systems process user input in a sequential order: domain is first predicted, then intent and slots are filled according to the semantic frames of the predicted domain. This pipeline approach, however, has many disadvantages including downstream error; i.e., if the system fails to predict the domain, then the system also fails to predict intent and slot. The main purpose of this paper is to mitigate the risk of downstream error propagation in traditional pipelined models and improve the predictive performance of domain, intent, and slot- all of which are critical steps for speech understanding and dialog systems- with a deep learning-based single joint model trained with an adversarial approach and long short-term memory (LSTM) algorithm. The systematic experimental analysis shows significant improvements in predictive performance for domain, intent, and entity with the proposed adversarial joint model, compared to the base joint model.

Acquaintance with Natural Language Processing for Building Smart Society

Natural Language Processing (NLP) deals with the spoken languages by using computer and Artificial Intelligence. As people from different regional areas using different digital platforms and expressing their views in their spoken language, it is now must to focus on working spoken languages in India to make our society smart and digital. NLP research grown tremendously in last decade which results in Siri, Google Assistant, Alexa, Cortona and many more automatic speech recognitions and understanding systems (ASR). Natural Language Processing can be understood by classifying it into Natural Language Generation and Natural Language Understanding. NLP is widely used in various domain such as Health Care, Chatbot, ASR building, HR, Sentiment analysis etc.

Speech Understanding System for Emotional Companion Robots

Abstract Within the project of emotional robot F-2 we develop a natural text parser for automatic speech comprehension. It is aimed at the construction of semantic representation and the selection of “scripts” – units for inference modelling and the selection of emotional reactions for the robot. The design of the speech understanding system follows the traditional concept of linguistic levels: it consequently constructs morphological, syntactic and semantic representations. Unlike neural networks, these representations are readable for a developer, so the accumulated data is available for statistical and analytical processing. Parser accepts written text after speech recognition (during actual talks with the robot) or during processing of everyday news and blogs from the internet. Parser may save text representations on each level of linguistic model to a database. In particular, semantic representations from daily internet processing are used as an accumulated knowledge database for the robot. In this paper we discuss the approach to the model of understanding through a competition of scripts on a robot companion.

Articulating Alternatives: Moving Past a Plug-and-Play Prosthetic Media Model

Articulating Alternatives: Moving Past a Plug-and-Play Prosthetic Media Model

Smart Enough to Talk With Us? Foundations and Challenges for Dialogue Capable AI Systems

Smart Enough to Talk With Us? Foundations and Challenges for Dialogue Capable AI Systems

The Item versus the Object in Memory: On the Implausibility of Overwriting As a Mechanism for Forgetting in Short-Term Memory.

The nature of forgetting in short-term memory remains a disputed topic, with much debate focussed upon whether decay plays a fundamental role (Berman et al., 2009; Altmann and Schunn, 2012; Barrouillet et al., 2012; Neath and Brown, 2012; Oberauer and Lewandowsky, 2013; Ricker et al., 2014) but much less focus on other plausible mechanisms. One such mechanism of long-standing in auditory memory is overwriting (e.g., Crowder and Morton, 1969) in which some aspects of a representation are “overwritten” and rendered inaccessible by the subsequent presentation of a further item. Here, we review the evidence for different forms of overwriting (at the feature and item levels) and examine the plausibility of this mechanism both as a form of auditory memory and when viewed in the context of a larger hearing, speech and language understanding system.

Exploiting Prosody for Syntactic Analysis in Automatic Speech Understanding

The relation between syntax and prosody is evident, even if the prosodic structure cannot be directly mapped to the syntactic one and vice versa. Syntax-to-prosody mapping is widely used in text-tospeech applications, but prosody-to-syntax mapping is mostly missing from automatic speech recognition/understanding systems. This paper presents an experiment towards filling this gap and evaluating whether a HMM-based automatic prosodic segmentation tool can be used to support the reconstruction of the syntactic structure directly from speech. Results show that up to 85% of syntactic clause boundaries and up to about 70% of embedded syntactic phrase boundaries could be identified based on the detection of phonological phrases. Recall rates do not depend further on syntactic layering, in other words, whether the phrase is multiply embedded or not. Clause boundaries can be well assigned to intonational phrase level in read speech and can be well separated from lower level syntactic phrases based on the type of the aligned phonological phrase(s). These findings can be exploited in speech understanding systems, allowing for the recovery of the skeleton of the syntactic structure, based purely on the speech signal.

Two-handed gesture recognition and fusion with speech to command a robot

Assistance is currently a pivotal research area in robotics, with huge societal potential. Since assistant robots directly interact with people, finding natural and easy-to-use user interfaces is of fundamental importance. This paper describes a flexible multimodal interface based on speech and gesture modalities in order to control our mobile robot named Jido. The vision system uses a stereo head mounted on a pan-tilt unit and a bank of collaborative particle filters devoted to the upper human body extremities to track and recognize pointing/symbolic mono but also bi-manual gestures. Such framework constitutes our first contribution, as it is shown, to give proper handling of natural artifacts (self-occlusion, camera out of view field, hand deformation) when performing 3D gestures using one or the other hand even both. A speech recognition and understanding system based on the Julius engine is also developed and embedded in order to process deictic and anaphoric utterances. The second contribution deals with a probabilistic and multi-hypothesis interpreter framework to fuse results from speech and gesture components. Such interpreter is shown to improve the classification rates of multimodal commands compared to using either modality alone. Finally, we report on successful live experiments in human-centered settings. Results are reported in the context of an interactive manipulation task, where users specify local motion commands to Jido and perform safe object exchanges.

English

A convenient and user -friendly interface for information retrieval through computer is an important issue. Speech being natural mode of communication among human beings has the potential of being a primary as well as convenient mode of interaction with computer. This is a good reason for building systems that communicate with users via speech. Marathi is one of the most widely used language is Maharashtra .An interesting domain for such speech understanding systems is information dialogs where the user wants to get some information by asking the system which takes the role of an Intelligent Machine in the field of interest. Speech recognition having much more application in health sector In order to make such a communication process possible we explore the new approach of Polly Clinic Inquiry System using Marathi Speech Recognition.

The Benefits of Remote Microphone Technology for Adults with Cochlear Implants

Cochlear implantation has become a standard practice for adults with severe to profound hearing loss who demonstrate limited benefit from hearing aids. Despite the substantial auditory benefits provided by cochlear implants, many adults experience difficulty understanding speech in noisy environments and in other challenging listening conditions such as television. Remote microphone technology may provide some benefit in these situations; however, little is known about whether these systems are effective in improving speech understanding in difficult acoustic environments for this population. This study was undertaken with adult cochlear implant recipients to assess the potential benefits of remote microphone technology. The objectives were to examine the measurable and perceived benefit of remote microphone devices during television viewing and to assess the benefits of a frequency-modulated system for speech understanding in noise. Fifteen adult unilateral cochlear implant users were fit with remote microphone devices in a clinical environment. The study used a combination of direct measurements and patient perceptions to assess speech understanding with and without remote microphone technology. The direct measures involved a within-subject repeated-measures design. Direct measures of patients' speech understanding during television viewing were collected using their cochlear implant alone and with their implant device coupled to an assistive listening device. Questionnaires were administered to document patients' perceptions of benefits during the television-listening tasks. Speech recognition tests of open-set sentences in noise with and without remote microphone technology were also administered. Participants showed improved speech understanding for television listening when using remote microphone devices coupled to their cochlear implant compared with a cochlear implant alone. This benefit was documented both when listening to news and talk show recordings. Questionnaire results also showed statistically significant differences between listening with a cochlear implant alone and listening with a remote microphone device. Participants judged that remote microphone technology provided them with better comprehension, more confidence, and greater ease of listening. Use of a frequency-modulated system coupled to a cochlear implant also showed significant improvement over a cochlear implant alone for open-set sentence recognition in +10 and +5 dB signal to noise ratios. Benefits were measured during remote microphone use in focused-listening situations in a clinical setting, for both television viewing and speech understanding in noise in the audiometric sound suite. The results suggest that adult cochlear implant users should be counseled regarding the potential for enhanced speech understanding in difficult listening environments through the use of remote microphone technology.

Semantic object synchronous understanding implemented with speech application language tags

A speech understanding system includes a language model comprising a combination of an N-gram language model and a context-free grammar language model. The language model stores information related to words and semantic information to be recognized. A module is adapted to receive input from a user and capture the input for processing. The module is further adapted to receive SALT application program interfaces pertaining to recognition of the input. The module is configured to process the SALT application program interfaces and the input to ascertain semantic information pertaining to a first portion of the input and output a semantic object comprising text and semantic information for the first portion by accessing the language model, wherein performing recognition and outputting the semantic object are performed while capturing continues for subsequent portions of the input.

Speech Understanding in Background Noise with the Two-Microphone Adaptive Beamformer BEAM™ in the Nucleus Freedom™ Cochlear Implant System

This paper evaluates the benefit of the two-microphone adaptive beamformer BEAM in the Nucleus Freedom cochlear implant (CI) system for speech understanding in background noise by CI users. A double-blind evaluation of the two-microphone adaptive beamformer BEAM and a hardware directional microphone was carried out with five adult Nucleus CI users. The test procedure consisted of a pre- and post-test in the lab and a 2-wk trial period at home. In the pre- and post-test, the speech reception threshold (SRT) with sentences and the percentage correct phoneme scores for CVC words were measured in quiet and background noise at different signal-to-noise ratios. Performance was assessed for two different noise configurations (with a single noise source and with three noise sources) and two different noise materials (stationary speech-weighted noise and multitalker babble). During the 2-wk trial period at home, the CI users evaluated the noise reduction performance in different listening conditions by means of the SSQ questionnaire. In addition to the perceptual evaluation, the noise reduction performance of the beamformer was measured physically as a function of the direction of the noise source. Significant improvements of both the SRT in noise (average improvement of 5-16 dB) and the percentage correct phoneme scores (average improvement of 10-41%) were observed with BEAM compared to the standard hardware directional microphone. In addition, the SSQ questionnaire and subjective evaluation in controlled and real-life scenarios suggested a possible preference for the beamformer in noisy environments. The evaluation demonstrates that the adaptive noise reduction algorithm BEAM in the Nucleus Freedom CI-system may significantly increase the speech perception by cochlear implantees in noisy listening conditions. This is the first monolateral (adaptive) noise reduction strategy actually implemented in a mainstream commercial CI.

Automatic Understanding of ATC Speech

In this paper we make a critical revision of the state-of-the-art in automatic speech processing as applied to air traffic control. We present the development of a new ATC speech understanding system comparing its performance and advantages to previously published experiences. The system has innovative solutions such as detecting the air/ground language spoken by air traffic controllers in an international airport with two official languages and the ability to adapt to new situations by automatically learning stochastic grammars from data, eliminating the need to write expensive and eternally incomplete grammars. A relevant new feature is the use of a speech understanding module able to extract semantically relevant information from the transcription of the sentences delivered by the speech recognizers. Two main assessment objectives are pursued and discussed throughout the paper: the effects of human spontaneity and the lack of linguistic coverage in understanding performance. The potential of this technology, ways of improvement, and proposals for the future are also presented

Toward Robust Speech Recognition and Understanding

The principal cause of speech recognition errors is a mismatch between trained acoustic/language models and input speech due to the limited amount of training data in comparison with the vast variation of speech. It is crucial to establish methods that are robust against voice variation due to individuality, the physical and psychological condition of the speaker, telephone sets, microphones, network characteristics, additive background noise, speaking styles, and other aspects. This paper overviews robust architecture and modeling techniques for speech recognition and understanding. The topics include acoustic and language modeling for spontaneous speech recognition, unsupervised adaptation of acoustic and language models, robust architecture for spoken dialogue systems, multi-modal speech recognition, and speech summarization. This paper also discusses the most important research problems to be solved in order to achieve ultimate robust speech recognition and understanding systems.

Fifty years of progress in speech understanding systems

Researchers working on human-machine interfaces realized nearly 50 years ago that automatic speech recognition (ASR) alone is not sufficient; one needs to impart linguistic knowledge to the system such that the signal could ultimately be understood. A speech understanding system combines speech recognition (i.e., the speech to symbols conversion) with natural language processing (i.e., the symbol to meaning transformation) to achieve understanding. Speech understanding research dates back to the DARPA Speech Understanding Project in the early 1970s. However, large-scale efforts only began in earnest in the late 1980s, with government research programs in the U.S. and Europe providing the impetus. This has resulted in many innovations including novel approaches to natural language understanding (NLU) for speech input, and integration techniques for ASR and NLU. In the past decade, speech understanding systems have become major building blocks of conversational interfaces that enable users to access and manage information using spoken dialogue, incorporating language generation, discourse modeling, dialogue management, and speech synthesis. Today, we are at the threshold of developing multimodal interfaces, augmenting sound with sight and touch. This talk will highlight past work and speculate on the future. [Work supported by an industrial consortium of the MIT Oxygen Project.]

The use of subword linguistic modeling for multiple tasks in speech recognition

Over the past several years, I have been conducting research on subword modeling in speech recognition. The research is most specifically aimed at the difficult task of identifying and characterizing unknown words, although the proposed framework also has utility in other recognition tasks such as phonological and prosodic modeling. The approach exploits the linguistic substructure of words by describing graphemic, phonemic, phonological, syllabic, and morphemic constraints through a set of context-free rules, and supporting the resulting parse trees with a corpus-trained probability model. A derived finite state transducer representation forms a natural means for integrating the trained model into a recognizer search. This paper describes several research projects I have been engaged in, together with my students and associates, aimed at exploring ways in which recognition tasks can benefit from such formal modeling of word substructure. These include phonological modeling, hierarchical duration modeling, sound-to-letter and letter-to-sound mapping, and automatic acquisition of unknown words in a speech understanding system. Results of several experiments in these areas are summarized here.

Brain, Meaning, Grammar, evolution

I reject Jackendoff's view of Universal Grammar as something that evolved biologically but applaud his integration of blackboard architectures. I thus recall the HEARSAY speech understanding system—the AI system that introduced the concept of “blackboard”—to provide another perspective on Jackendoff's architecture.

문장음성 이해를 위한 확률모델에 관한 연구

본 논문에서는 사전과 시소러스를 이용하여 문장음성 이해를 위한 확률모델을 제안한다. 제안한 확률모델은 입력되는 음성과 텍스트 문장에서 단어를 추출한다. 컴퓨터가 선택한 카테고리의 사전 DB와 입력된 문장에서 추출된 단어와 비교하고 확률모델로부터 확률값을 얻는다. 이때 컴퓨터로부터 상위어 정보를 알아내고 상위어 사전을 검색하여 단어를 추출하고 입력된 단어와 확률 모델을 비교하여 결과값을 얻는다. 사전과 상위어 사전으로부터 얻은 두개의 확률값을 더하고 그 값을 미리 정해진 임계값과 비교하여 문장의 이해도를 측정한다. 이와 같은 이해 시스템을 스무고개 게임에 적용시켜 그 성능을 평가 하였다. 상위어 확률 값(<TEX>$\alpha$</TEX>)이 0.9이고 임계값 (<TEX>$\beta$</TEX>)은 0.38일 때 문장음성 이해의 정확도는 79.8%였다. In this paper, we propose a stochastic model for sentence speech understanding using dictionary and thesaurus. The proposed model extracts words from an input speech or text into a sentence. A computer is sellected category of dictionary database compared the word extracting from the input sentence calculating a probability value to the compare results from stochastic model. At this time, computer read out upper dictionary information from the upper dictionary searching and extracting word compared input sentence caluclating value to the compare results from stochastic model. We compare adding the first and second probability value from the dictionary searching and the upper dictionary searching with threshold probability that we measure the sentence understanding rate. We evaluated the performance of the sentence speech understanding system by applying twenty questions game. As the experiment results, we got sentence speech understanding accuracy of 79.8%. In this case, probability (<TEX>$\alpha$</TEX>) of high level word is 0.9 and threshold probability (<TEX>$\beta$</TEX>) is 0.38.

Diagnostic system for speech articulation and speech understanding

With the increase in access to multimedia computers, speech training can be made available to patients with no continuous assistance required from speech therapists. Another function the system can easily perform is screening testing of speech fluency providing directed information to patients who have various speech disorders and problems with understanding speech. The idea underlying the proposed system is a programmed speech therapy training algorithm consisting of diagnostic tools and rehabilitation devices connected with it. The first function the system described in the paper has to perform is data acquisition where information about the patient’s medical history is collected. This is done through electronic questionnaires. The next function is analysis of the speech signal articulated by the patient when prompted by the computer followed by some multimedia tests carried out in order to assess the subject’s ability to understand speech. Next, the results of the electronic questionnaire, the patient’s voice, and the patients’ reactions are automatically analyzed, based on that the system automatically diagnoses possible speech disorders and how strong they are. A large number of schoolchildren were tested with this method. In the paper foundations of applied speech testing method and obtained results will be demonstrated.