Speech Dialogue System Research Articles

Realization of Chinese-English Bilingual Speech Dialogue System using Machine Translation Technology

The realization of a Chinese-English bilingual speech dialogue system through machine translation technology involves developing a sophisticated system capable of seamlessly translating spoken language between Chinese and English in real-time. This system employs cutting-edge machine learning algorithms, neural networks, and natural language processing techniques to accurately interpret and translate speech inputs from one language to another. By integrating advanced speech recognition and translation models, users can engage in fluid and natural conversations across language barriers, opening up new possibilities for cross-cultural communication and interaction. This paper introduces a Statistical Phase-based Bilingual Speech (SPBS) system designed to facilitate seamless language translation and dialogue between multiple languages, with a focus on Chinese and English. Leveraging advanced machine learning models and techniques, such as Recurrent Neural Networks (RNN) with Bidirectional Long Short-Term Memory (Bi-LSTM) architecture, the SPBS system achieves high translation accuracy, computational efficiency, and fluency of translations. The system's multilingual model attains an impressive translation accuracy of 97% while processing 10 sentences per second, with positive feedback on the fluency of translations. Trained on a substantial dataset of 1 million bilingual sentence pairs, the SPBS model maintains a compact size of 500 MB. Furthermore, the paper presents the machine learning settings and training progress of the SPBS system, demonstrating its effectiveness in accurately classifying and translating speech inputs across languages. The system's multilingual model attains an impressive translation accuracy of 97% while processing 10 sentences per second, with positive feedback on the fluency of translations. Trained on a substantial dataset of 1 million bilingual sentence pairs, the SPBS model maintains a compact size of 500 MB.

Human-computer interaction for virtual-real fusion

Human-computer interaction for virtual-real fusion

Human-Computer Interaction System: A Survey of Talking-Head Generation

Virtual human is widely employed in various industries, including personal assistance, intelligent customer service, and online education, thanks to the rapid development of artificial intelligence. An anthropomorphic digital human can quickly contact people and enhance user experience in human–computer interaction. Hence, we design the human–computer interaction system framework, which includes speech recognition, text-to-speech, dialogue systems, and virtual human generation. Next, we classify the model of talking-head video generation by the virtual human deep generation framework. Meanwhile, we systematically review the past five years’ worth of technological advancements and trends in talking-head video generation, highlight the critical works and summarize the dataset.

How can I help you? Design principles for task-oriented speech dialog systems in customer service

Organizations are increasingly delegating customer inquiries to speech dialog systems (SDSs) to save personnel resources. However, customers often report frustration when interacting with SDSs due to poorly designed solutions. Despite these issues, design knowledge for SDSs in customer service remains elusive. To address this research gap, we employ the design science approach and devise a design theory for SDSs in customer service. The design theory, including 14 requirements and five design principles, draws on the principles of dialog theory and undergoes validation in three iterations using five hypotheses. A summative evaluation comprising a two-phase experiment with 205 participants yields positive results regarding the user experience of the artifact. This study contributes to design knowledge for SDSs in customer service and supports practitioners striving to implement similar systems in their organizations.

Voice To Text Using ASR and HMM

Abstract: In the modern world, communication technologies have been integrated with the internet. SDSs (speech dialog systems) use voice search technology to answer user’s queries with the information they request. Information normally is stored in a large database, and a query needs to be matched with a field in the database. Research on acoustic modeling focuses on improving robustness to environmental noise, channel conditions, and speaker variation, while research on pronunciation addresses the issue of unseen word sounds and variability in pronunciation.

Speaker extraction network with attention mechanism for speech dialogue system

Speaker extraction network with attention mechanism for speech dialogue system

Adversarial Training for Multi Domain Dialog System

Natural Language Understanding and Speech Understanding systems are now a global trend, and with the advancement of artificial intelligence and machine learning techniques, have drawn attention from both the academic and business communities. Domain prediction, intent detection and entity extraction or slot fillings are the most important parts for such intelligent systems. Various traditional machine learning algorithms such as Bayesian algorithm, Support Vector Machine, and Artificial Neural Network, along with recent Deep Neural Network techniques, are used to predict domain, intent, and entity. Most language understanding systems process user input in a sequential order: domain is first predicted, then intent and slots are filled according to the semantic frames of the predicted domain. This pipeline approach, however, has many disadvantages including downstream error; i.e., if the system fails to predict the domain, then the system also fails to predict intent and slot. The main purpose of this paper is to mitigate the risk of downstream error propagation in traditional pipelined models and improve the predictive performance of domain, intent, and slot- all of which are critical steps for speech understanding and dialog systems- with a deep learning-based single joint model trained with an adversarial approach and long short-term memory (LSTM) algorithm. The systematic experimental analysis shows significant improvements in predictive performance for domain, intent, and entity with the proposed adversarial joint model, compared to the base joint model.

Improved Speech Command Classification System for Sinhala Language based on Automatic Speech Recognition

The recent advancements in conversational Artificial Intelligence (AI) are fastly getting integrated with every realm of human lives. Conversational agents who can learn, understand human languages and mimic the human thinking process have already created a revolution in human lifestyle. Understanding the intention of a speaker from his natural speech is a significant step in conversational AI. A major challenge that hinders the efficacy of this process is the lack of language resources. In this research, we address this issue and develop a domain-specific speech command classification system for the Sinhala language, one of the low-resourced languages. An effective speech command classification system can be utilized in several value added applications such as speech dialog systems. Our speech command classification system is developed by integrating Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). The ASR engine is implemented using Gaussian Mixture Model-Hidden Markov Model (GMM-HMM) and it converts a Sinhala speech command into a corresponding text representation. The text classifier, which is implemented as an ensemble unit of several classifiers, predicts the intent of the speaker when provided with the above text output. In this paper, we discuss and evaluate various algorithms and techniques that can be utilized to optimize the performance of both the ASR and text classifier. As well, we present our novel Sinhala speech data corpus of 4.15[Formula: see text]h which is based on the banking domain. As the final outcome, our system reports its Sinhala speech command classification accuracy as 91.03%. It shows that our system outperforms the state-of-the-art speech-to-intent mapping systems developed for the Sinhala language. The individual evaluation on the ASR system reports a 9.91% Word Error Rate and a 19.95% Sentence Error Rate, suggesting the applicability of advanced speech recognition techniques despite the limited language resources. Finally, our findings deliver useful insights on further research in speech command classification in the low-resourced context.

Automatic recognition of negative emotion in speech using support vector machine

This paper addresses negative emotion recognition using paralinguistic information in speech for speech dialogue system. Speech conveys not only linguistic information but also paralinguistic and non-linguistic information such as the emotions, attitudes, and intentions. This easily perceivable information plays a key role in a spoken dialog system. However, most of previous speech recognition systems fail to consider this significant information, focusing only on linguistic information, thus hindering the development of more natural speech dialog systems. In order to utilize these significant information for speech dialog systems, this paper focuses on negative emotion recognition from Japanese utterances. 6552-dimensional acoustic features were extracted from 6300 Japanese utterances of 50 people in three emotional state: negative; positive; and neutral. Negative emotion includes anger, sad and dislike. While positive emotion includes favor, joy, and relief. They were classified by SVM and evaluated by a 10-fold cross validation. The experimental result showed the recognition rate of 93.4 % for the classification of negative and positive and 95.0% for the classification of negative and neutral.

Signal processing techniques for seat belt microphone arrays

Microphones integrated on a seat belt are an interesting alternative to conventional sensor positions used for hands-free telephony or speech dialog systems in automobile environments. In the setup presented in this contribution, the seat belt consists of three microphones which usually lay around the shoulder and chest of a sitting passenger. The main benefit of belt microphones is the small distance from the talker’s mouth to the sensor. As a consequence, an improved signal quality in terms of a better signal-to-noise ratio (SNR) compared to other sensor positions, e.g., at the rear view mirror, the steering wheel, or the center console, can be achieved. However, the belt microphone arrangement varies considerably due to movements of the passenger and depends on the size of the passenger. Furthermore, additional noise sources arise for seat belt microphones: they can easily be touched, e.g., by clothes, or might be in the path of an air-stream from the automotive ventilation system. This contribution presents several robust signal enhancement algorithms designed for belt microphones in multi-seat scenarios. The belt microphone with the highest SNR (usually closest to the speaker’s mouth) is selected for speech signal enhancement. Further improvements can be achieved if all belt microphone signals are combined to a single output signal. The proposed signal enhancement system for belt microphones includes a robust echo cancelation scheme, three different microphone combining approaches, a sophisticated noise estimation scheme to track stationary as well as non-stationary noise, and a speech mixer to combine the signals from each seat belt to a single channel output in a multi-seat scenario.

Evaluation of speech-based HMI concepts for information exchange tasks: A driving simulator study

Due to the mobile Internet revolution, people tend to browse the Web while driving their car which puts the driver's safety at risk. Therefore, an intuitive and non-distractive in-car speech interface to the Web needs to be developed. Before developing a new speech dialog system (SDS) in a new domain developers have to examine the user's preferred interaction style and its influence on driving safety.This paper reports a driving simulation study, which was conducted to compare different speech-based in-car human–machine interface concepts concerning usability and driver distraction. The applied SDS prototypes were developed to perform an online hotel booking by speech while driving. The speech dialog prototypes were based on different speech dialog strategies: a command-based and a conversational dialog. Different graphical user interface (GUI) concepts (one including a human-like avatar) were designed in order to support the respective dialog strategy the most and to evaluate the effect of the GUI on usability and driver distraction.The results show that only few differences concerning speech dialog quality were found when comparing the speech dialog strategies. The command-based dialog was slightly better accepted than the conversational dialog, which seems to be due to the high concept error rate of the conversational dialog. A SDS without GUI also seems to be feasible for the driving environment and was accepted by the users. The comparison of speech dialog strategies did not reveal differences in driver distraction. However, the use of a GUI impaired the driving performance and increased gaze-based distraction. The presence of an avatar was not appreciated by participants and did not affect the dialog performance. Concerning driver distraction, the virtual agent did neither negatively affect the driving performance nor increase visual distraction.The results implicate that in-car SDS developers should take both speaking styles into consideration when designing an SDS for information exchange tasks. Furthermore, developers have to consider reducing the content presented on the screen in order to reduce driver distraction. A human-like avatar was not appreciated by users while driving. Research should further investigate if other kinds of avatars might achieve different results.

Проектирование систем речевого диалога

The basic problem we address during this study is to investigate by the Kellerman-Plutchick Life Style Index the characteristics of psychological defense in the youth, concentrating on determining the average activation level and searching for the ways of changing the intensity of different defense mechanisms. We analyze specific characteristic features of psychological defense mechanisms among the youth and their correlations with different psychological features (personal traits, temperament, accentuations, risk inclination, new sensation seeking) and create the psychological portraits of young persons, specific for each of the basic mechanisms of psychological defense. We place high emphasis on risk behavior analysis. Mathematical statistics techniques (primary statistics, Student’s ttest, Pearson’s r, factor analysis, linear regressions, multivariate regressions) which have been used for data processing are also described in the article.

A dynamic multi-channel speech enhancement system for distributed microphones in a car environment

Supporting multiple active speakers in automotive hands-free or speech dialog applications is an interesting issue not least due to comfort reasons. Therefore, a multi-channel system for enhancement of speech signals captured by distributed distant microphones in a car environment is presented. Each of the potential speakers in the car has a dedicated directional microphone close to his position that captures the corresponding speech signal. The aim of the resulting overall system is twofold: On the one hand, a combination of an arbitrary pre-defined subset of speakers’ signals can be performed, e.g., to create an output signal in a hands-free telephone conference call for a far-end communication partner. On the other hand, annoying cross-talk components from interfering sound sources occurring in multiple different mixed output signals are to be eliminated, motivated by the possibility of other hands-free applications being active in parallel. The system includes several signal processing stages. A dedicated signal processing block for interfering speaker cancellation attenuates the cross-talk components of undesired speech. Further signal enhancement comprises the reduction of residual cross-talk and background noise. Subsequently, a dynamic signal combination stage merges the processed single-microphone signals to obtain appropriate mixed signals at the system output that may be passed to applications such as telephony or a speech dialog system. Based on signal power ratios between the particular microphone signals, an appropriate speaker activity detection and therewith a robust control mechanism of the whole system is presented. The proposed system may be dynamically configured and has been evaluated for a car setup with four speakers sitting in the car cabin disturbed in various noise conditions.

Impact of cognitive load and frustration on drivers’ speech.

Secondary tasks such as cell phone calls or interaction with automated speech dialog systems (SDSs) increase the driver’s cognitive load as well as the probability of driving errors. This study analyzes speech production variations due to cognitive load and emotional state of drivers in real driving conditions. Speech samples were acquired from 24 female and 17 male subjects (approximately 8.5 h of data) while talking to a co-driver and communicating with two automated call centers, with emotional states (neutral, negative) and the number of necessary SDS query repetitions also labeled. A consistent shift in a number of speech production parameters (pitch, first format center frequency, spectral center of gravity, spectral energy spread, and duration of voiced segments) was observed when comparing SDS interaction against co-driver interaction; further increases were observed when considering negative emotion segments and the number of requested SDS query repetitions. A mel frequency cepstral coefficient based Gaussian mixture classifier trained on 10 male and 10 female sessions provided 91% accuracy in the open test set task of distinguishing co-driver interactions from SDS interactions, suggesting—together with the acoustic analysis—that it is possible to monitor the level of driver distraction directly from their speech.

An Integrated Natural Language Processing Approach for Conversation System

Abstract —The main aim of this research is to investigate anovel technique for implementing a more natural and intelligentconversation system. Conversation systems are designed to converselike a human as much as their intelligent allows. Sometimes, we canthink that they are the embodiment of Turing’s vision. It usually toreturn a predetermined answer in a predetermined order, but con-versations abound with uncertainties of various kinds. This researchwill focus on an integrated natural language processing approach.This approach includes an integrated knowledge-base constructionmodule, a conversation understanding and generator module, anda state manager module. We discuss effectiveness of this approachbased on an experiment. Keywords —Conversation System; integrated knowledge-base con-struction; conversation understanding and generator; state manager. I. I NTRODUCTION A Conversation system is a computer system intended toconverse with a human, with a coherent structure. Con-versation systems have employed text, image, speech, emotion,gestures and other modes for communication on both the inputand output channel.Man-machine-conversation system has become more im-portant in 21 century. Several speech dialogue systems andweb agent have been developed in order to provide a veryeaseful interaction for dissimilar users. The primary aim isto satisfy the user without or at least with reduced humaninvolvement [1]. A few well known scientists believe thatcomputers will achieve capabilities comparable to humanreasoning and understanding of languages by 2020 [2]. Wethink a more humanoid interactive system will be achievedalong with recent advances in Natural Language Processing(NLP) and Artificial Intelligence (AI) field.II. R

Multimodal identity tracking in a smart room

The automatic detection, tracking, and identification of multiple people in intelligent environments are important building blocks on which smart interaction systems can be designed. Those could be, e.g., gesture recognizers, head pose estimators or far-field speech recognizers and dialog systems. In this paper, we present a system which is capable of tracking multiple people in a smart room environment while inferring their identities in a completely automatic and unobtrusive way. It relies on a set of fixed and active cameras to track the users and get close-ups of their faces for identification, and on several microphone arrays to determine active speakers and steer the attention of the system. Information coming asynchronously from several sources, such as position updates from audio or visual trackers and identification events from identification modules, is fused at higher level to gradually refine the room's situation model. The system has been trained on a small set of users and showed good performance at acquiring and keeping their identities in a smart room environment.

Multilingual spoken language processing

In the past decade, the performance of spoken language understanding systems has improved dramatically, including speech recognition, dialog systems, speech summarization, and text and speech translation. This has resulted in an increasingly widespread use of speech and language technologies in a wide variety of applications. With more than 6,900 languages in the world and the current trend of globalization, one of the most important challenges in spoken language technologies today is the need to support multiple input and output languages, especially if applications are intended for international markets, linguistically diverse user communities, and nonnative speakers. In many cases these applications have to support even multiple languages simultaneously to meet the needs of a multicultural society. Consequently, new algorithms and tools are required that support the simultaneous recognition of mixed-language input, the summarization of multilingual text and spoken documents, the generation of output in the appropriate language, or the accurate translation from one language to another. This article surveys significant ongoing research programs as well as trends, prognoses, and open research issues with a special emphasis on multilingual speech processing as described in detail in the work of Schultz and Hirschberg (2006) and multilingual language processing as presented in the work of Fung (2006).

Dialogue as a basis for construction of speech systems

The structure of speech dialogue systems is analyzed. It is shown that the creation of speech recognition and synthesis systems is connected with the solution of direct and inverse problems as applied to the organization of speech dialogue. Attention is paid to the multilevel hierarchical organization of speech dialogue systems in which the interaction between recognition and synthesis channels is realized through a common knowledgebase. A generalized structure of data representation at different hierarchy levels is analyzed, and the hierarchy of qualities determining properties of a dialogue system is considered.

Confidence measure and rejection based on correctness probability of recognition candidates

We propose a confidence measure expressing the correctness probability of recognition candidates, to be applied to a speech dialog system using a dialog strategy based on the degree of confidence o...

Natural speech dialogue systems

The application of speech recognition and understanding to real-life situations generates new scientific problems. To illustrate the latter, the example of an automatic switchboard system is given that can be used in a very natural communication style. It is based on a high-end speech recognizer that outputs a word lattice which is subject to further evaluation by a speech understanding and a dialogue component. Although word error rates are above 20% for this task, the integration of additional knowledge sources like the telephone directory and the dialogue history can add to both the acoustic and the language model in order to achieve highly improved performance. As recognition is not perfect, the user has to confirm what the system understood. In order to avoid lengthy confirmations, it was requested to base the level of confirmation (repeated question, explicit or implicit confirmation) on the probability that the request was understood correctly. This was formulated within the statistical framework of speech understanding and successfully implemented into our prototype system. The degree of reliability controls the further dialogue flow. Other useful applications of this reliability measure will be described.