Dialogue Management Research Articles

Combining speech-based and linguistic classifiers to recognize emotion in user spoken utterances

Abstract In this paper we propose to combine speech-based and linguistic classification in order to obtain better emotion recognition results for user spoken utterances. Usually these approaches are considered in isolation and even developed by different communities working on emotion recognition and sentiment analysis. We propose modeling the users emotional state by means of the fusion of the outputs generated with both approaches, taking into account information that is usually neglected in the individual approaches such as the interaction context and errors, and the peculiarities of transcribed spoken utterances. The fusion approach allows to employ different recognizers and can be integrated as an additional module in the architecture of a spoken conversational agent, using the information generated as an additional input for the dialog manager to decide the next system response. We have evaluated our proposal using three emotionally-colored databases and obtained very positive results.

Quantifying local community voices in the decision-making process: insights from the Mount Cameroon National Park REDD+ project

ABSTRACTOne key component in community development is the ability to negotiate and make contentious environmental decisions that have sustainable environmental and livelihoods benefits. This study examines the community voice during the consultation and decision-making process to evaluate if the inclusion of community delegates is meant to legitimise REDD+ or leads to community development. Likert-scale questionnaire, focus-group discussions and interviews were used to collect data from 259 respondents located in 12 villages around Mount Cameroon National Park (MCNP). Results show that the level of local participation in the MCNP-REDD+ project directly relates to both standing and influence granted to the local stakeholders. Although MCNP managers understood community concerns, these concerns are not being addressed. Local community delegates are granted access to discussion forums, but they are not influencing the decision-making process or the design of REDD+ within MCNP. The voice of all stakeholders must be heard to ensure equitable development and socio-environmental legitimacy of projects. A decision-making process that is transparent and familiar to local communities will enhance their confidence to engage in debates as equal stakeholders and ensure that decisions reflect community consent. Capacity building in the decision-making process will empower the community to efficiently engage in consultation, dialogue and transparent forest management decisions.

Principles of performance dialogue in public administration

PurposeThe literature acknowledges the importance of interpretative processes, discussion, and organizational learning in public performance management, but a knowledge gap remains concerning the mechanism of performance dialogue. To fill this gap, the purpose of this paper is to study the principles of performance dialogue and collaborative performance management in public administration.Design/methodology/approachThe study utilizes a longitudinal research setting and analyzes the evolution of performance management practices in one city organization in Finland.FindingsThe study suggests that performance dialogue needs to be integrated with management practices and explains how this can be done. Three guiding principles of performance dialogue and collaborative performance management are derived. These underline the role of “referees of the information game”, a supportive and encouraging environment and a focus on the use of performance information.Practical implicationsPerformance information is too often provided as a back-office function, and dialogue with information users is either completely lacking or somehow disturbed. The performance dialogue provides a platform for collaborative sense making and helps managers to better understand the complex phenomena and processes they are responsible for.Originality/valueThe literature dealing specifically with the change from centralized and vertical performance management practices toward decentralized and horizontal practices is still scarce. This paper provides a new perspective on management control and organizational learning in public administration based on performance dialogue.

Using Vision and Speech Features for Automated Prediction of Performance Metrics in Multimodal Dialogs

Predicting and analyzing multimodal dialog user experience (UX) metrics, such as overall call experience, caller engagement, and latency, among other metrics, in an ongoing manner is important for evaluating such systems. We investigate automated prediction of multiple such metrics collected from crowdsourced interactions with an open-source, cloud-based multimodal dialog system in the educational domain. We extract features from both the audio and video signals and examine the efficacy of multiple machine learning algorithms in predicting these performance metrics. The best performing audio features consist of multiple low-level audio descriptors—intensity, loudness, cepstra, pitch, and so on—and their functionals, extracted using the OpenSMILE toolkit, while the video features are bags of visual words that use 3D Scale-Invariant Feature Transform descriptors. We find that our proposed methods outperform the majority vote classification baseline in predicting various UX metrics rated by both the user and experts. Our results suggest that such automated prediction of performance metrics can not only inform the qualitative and quantitative analysis of dialogs but also be potentially incorporated into dialog management routines for positively impacting UX and other metrics during the course of the interaction.

Developing a Conversational Virtual Standardized Patient to Enable Students to Practice History-Taking Skills.

Although traditional virtual patient simulations are designed to teach and assess clinical reasoning skills, few employ conversational dialogue with the patients. The virtual standardized patients (VSPs) described herein represent standardized patients that students interview using natural language. Students take histories and develop differential diagnoses of the VSPs as much as they would with standardized or actual patients. The student-VSP interactions are recorded, creating a comprehensive record of questions and the order in which they were asked, which can be analyzed to assess information-gathering skills. Students document the encounter in an electronic medical record created for the VSPs. The VSP was developed by integrating a dialogue management system (ChatScript) with emotionally responsive 3D characters created in a high-fidelity game engine (Unity). The system was tested with medical students at the Ohio State University College of Medicine. Students are able to take a history of a VSP, develop a differential diagnosis, and document the encounter in the electronic medical record. Accuracy of the VSP responses ranged from 79% to 86%, depending on the complexity of the case, type of history obtained, and skill of the student. Students were able to accurately develop an appropriate differential diagnosis on the basis of the information provided by the patient during the encounter. The VSP enables students to practice their history-taking skills before encounters with standardized or actual patients. Future developments will focus on creating an assessment module that will automatically analyze VSP sessions and provide immediate student feedback.

A Conventional Dialogue Model Based on Dialogue Patterns

Our work aims at designing a dialogue manager dedicated to agents that interact with humans. In this article, we investigate how dialogue patterns at the dialogue act level extracted from Human-Human interactions can be fruitfully used by a software agent to interact with a human.We show how these patterns can be leveraged via a dialogue game structure in order to benefit to the dialogue management process of an agent. We describe how empirically specified dialogue games can be employed on both interpretative and generative levels of dialogue management. We present Dogma, an open-source module that can be used by an agent to manage its conventional communicative behaviour. We show that our library of dialogue games can be used into Dogma to generate fragments of dialogue that are strongly coherent from a human perspective.

A User-Centric Platform PRINTEPS to Develop Integrated Intelligent Applications and Application to Robot Teahouse

We are developing PRactical INTEligent aPplicationS (PRINTEPS), which is a user-centric platform to develop integrated intelligent applications only by combining four types of modules such as knowledge-based reasoning, speech dialog, image sensing and motion management. PRINTEPS supports end users to participate in AI applications design (user participation design) and to develop applications easily. This paper introduces the architecture and applications of PRINTEPS for robot teahouse robot where multiple people and robots cooperate.

Dialogue manager domain adaptation using Gaussian process reinforcement learning

Spoken dialogue systems allow humans to interact with machines using natural speech. As such, they have many benefits. By using speech as the primary communication medium, a computer interface can facilitate swift, human-like acquisition of information. In recent years, speech interfaces have become ever more popular, as is evident from the rise of personal assistants such as Siri, Google Now, Cortana and Amazon Alexa. Recently, data-driven machine learning methods have been applied to dialogue modelling and the results achieved for limited-domain applications are comparable to or out-perform traditional approaches. Methods based on Gaussian processes are particularly effective as they enable good models to be estimated from limited training data. Furthermore, they provide an explicit estimate of the uncertainty which is particularly useful for reinforcement learning. This article explores the additional steps that are necessary to extend these methods to model multiple dialogue domains. We show that Gaussian process reinforcement learning is an elegant framework that naturally supports a range of methods, including prior knowledge, Bayesian committee machines and multi-agent learning, for facilitating extensible and adaptable dialogue systems.

From VoiceXML to multimodal mobile Apps: development of practical conversational interfaces

Speech Technologies and Language Processing have made possible the development of a number of new applications which are based on conversational interfaces. In this paper, we describe two approaches to bridge the gap between the academic and industrial perspectives in order to develop conversational interfaces using an academic paradigm for dialog management while employing the industrial standards. The advances in these technologies have made possible to extend the initial applications of conversational interfaces from only spoken interaction (for instance, by means of VoiceXML-based systems) to multimodal services by means of mobile devices (for instance, using the facilities provided by the Android OS). Our proposal has been evaluated with the successful development of different spoken and multimodal conversational interfaces.

A proposal to manage multi-task dialogs in conversational interfaces

The emergence of smart devices and recent advances in spoken language technology are currently extending the use of conversational interfaces and spoken interaction to perform many tasks. The dialog management task of a conversational interface consists of selecting the next system response considering the user's actions, the dialog history, and the results of accessing the data repositories. In this paper we describe a dialog management technique adapted to multi-task conversational systems. In our proposal, specialized dialog models are used to deal with each specific subtask of dialog objective for which the dialog system has been designed. The practical application of the proposed technique to develop a dialog system acting as a customer support service shows that the use of these specialized dialog models increases the quality and number of successful interactions with the system in comparison with developing a single dialog model.

Conversational In-Vehicle Dialog Systems: The past, present, and future

Automotive technology rapidly advances with increasing connectivity and automation. These advancements aim to assist safe driving and improve user travel experience. Before the realization of a full automation, in-vehicle dialog systems may reduce the driver distraction from many services available through connectivity. Even when a full automation is realized, in-vehicle dialog systems still play a special role in assisting vehicle occupants to perform various tasks. On the other hand, in-vehicle use cases need to address very different user conditions, environments, and industry requirements than other uses. This makes the development of effective and efficient in-vehicle dialog systems challenging; it requires multidisciplinary expertise in automatic speech recognition, spoken language understanding, dialog management (DM), natural language generation, and application management, as well as field system and safety testing. In this article, we review research and development (R&D) activities for in-vehicle dialog systems from both academic and industrial perspectives, examine findings, discuss key challenges, and share our visions for voice-enabled interaction and intelligent assistance for smart vehicles over the next decade.

Evaluation of natural language understanding based speech dialog interface's effectiveness regarding car navigation system usability performance

Recently, new cell phone services enable taking input by speaking to a quasi-agent using highly accurate speech recognition technologies. However, there are two problems when equipping a vehicle with these technologies. First, we do not understand yet the effect on driver’s usability performance of a car navigation system equipped with spoken dialogue interface. Second, the corresponding design techniques to obtain sufficiently good navigation results has not yet been determined. In this study, we first discuss problems involving speech recognition technologies and the types of speech interface used in this study. Second, we experimentally compare in pseudo-driving conditions the effect on driver’s physiology of various speech user interfaces: the Command Input (CI) interface, the Natural Language Understanding (NLU) interface, and the NLU/Spoken Dialogue Management (SDM) interface. According to the analysis of physiological signals, parasympathetic nervous system activity index value was higher when using the NLU/SDM interface than when using the CI interface. Furthermore, both subjective assessment result and time satisfaction degree of NLU/SDM interface were higher than others. Therefore, we can conclude that the NLU/SDM-type interface has better usability performance.

A New Perspective of Negotiation-Based Dialog to Enhance Metacognitive Skills in the Context of Open Learner Models

Negotiation mechanism using conversational agents (chatbots) has been used in Open Learner Models (OLM) to enhance learner model accuracy and provide opportunities for learner reflection. Using chatbots that allow for natural language discussions has shown positive learning gains in students. Traditional OLMs assume a learner to be able to manage their own learning and already in an appropriate affective/behavioral state that is conducive for learning. This paper proposes a new perspective of learning that advances the state of the art in fully-negotiated OLMs by exploiting learner’s affective & behavioral states to generate engaging natural language dialogues that train them to enhance their metacognitive skills. In order to achieve this, we have developed the NDLtutor that provides a natural language interface to learners. Our system generates context-aware dialogues automatically to enhance learner participation and reflection. This paper provides details on the design and implementation of the NDLtutor and discusses two evaluation studies. The first evaluation study focuses on the dialogue management capabilities of our system and demonstrates that our dialog system works satisfactorily to realize meaningful and natural interactions for negotiation. The second evaluation study investigates the effects of our system on the self-assessment and self-reflection of the learners. The results of the evaluations show that the NDLtutor is able to produce significant improvements in the self-assessment accuracy of the learners and also provides adequate support for prompting self-reflection in learners.

Spoken dialog systems based on online generated stochastic finite-state transducers

In this paper, we present an approach for the development of spoken dialog systems based on the statistical modelization of the dialog manager. This work focuses on three points: the modelization of the dialog manager using Stochastic Finite-State Transducers, an unsupervised way to generate training corpora, and a mechanism to address the problem of coverage that is based on the online generation of synthetic dialogs. Our proposal has been developed and applied to a sport facilities booking task at the university. We present experimentation evaluating the system behavior on a set of dialogs that was acquired using the Wizard of Oz technique as well as experimentation with real users. The experimentation shows that the method proposed to increase the coverage of the Dialog System was useful to find new valid paths in the model to achieve the user goals, providing good results with real users.

A framework for improving error detection and correction in spoken dialog systems

Despite the recent improvements in performance and reliably of the different components of dialog systems, it is still crucial to devise strategies to avoid error propagation from one another. In this paper, we contribute a framework for improved error detection and correction in spoken conversational interfaces. The framework combines user behavior and error modeling to estimate the probability of the presence of errors in the user utterance. This estimation is forwarded to the dialog manager and used to compute whether it is necessary to correct possible errors. We have designed an strategy differentiating between the main misunderstanding and non-understanding scenarios, so that the dialog manager can provide an acceptable tailored response when entering the error correction state. As a proof of concept, we have applied our proposal to a customer support dialog system. Our results show the appropriateness of our technique to correctly detect and react to errors, enhancing the system performance and user satisfaction.

Structural Optimization and Online Evolutionary Learning for Spoken Dialog Management

Designing dialog management (DM) policies that are robust to environmental noises is a nontrivial task. Approaches based on reinforcement learning (RL) are popular in academia and have been empirically shown to exhibit much better performance than handcrafted policies. However, the policies trained using RL are mostly incomprehensible, thus limiting the deployments for commercial applications. Policy optimization using genetic algorithm (GA) is a relatively new approach to spoken DM. The most notable advantage of this approach is that the trained policies can be directly interpreted by human experts. In this letter, we make several contributions to the GA-based framework. First, a structural policy learning procedure is presented. Second, a new fitness estimation method based on fitted policy evaluation is proposed. Finally, combining with these methods, an online evolutionary policy learning algorithm is designed which is much more data efficient than direct policy search using Monte Carlo simulations. These proposed approaches are empirically evaluated and compared with several state-of-the-art methods in a simulated environment. The experiments show favorable results for our approach.

A novel factored POMDP model for affective dialogue management

Partially observable Markov decision process (POMDP) model has been demonstrated many times to be suited for robust spoken dialogue management. Recently, some factored representations of POMDP model are designed for specific dialogue tasks. This paper proposes a novel factored POMDP model to describe a new application of affective dialogue management. Different from existing models, the user's state space and the system's observation space are both divided into two distinct components: goal and emotion. Moreover, the system's action space is for the first time factored into two parts, i.e., goal response and emotion response, and the reward function is accordingly updated by weighted sum of the two-part rewards. An example of intelligent music player is given to explain how to apply the new model to build an affective dialogue system. Four experiments are designed to reveal the influence of key parameters on the system performance. The simulation results demonstrate the rationality and feasibility of the proposed model.

A Novel Model for QUestion Answering Railway System: QUARS

Objectives: The plethora of information available on the web makes it difficult to search for a specific and suitable query. So, Question Answering dialogue system becomes helpful to provide precise and short answers to users instead of lengthy documents or passages. Method: The main focus is on keyword based approach and partial linguistic analysis to process the query. The main component is Dialogue Manager which clarifies the conversation between user and system to resolve the elliptical and anaphoric problems. Findings/ Application: The paper is based upon the development of Question answering dialogue system in Hindi language for the domain of Railways. The study is restricted to Northern India alone. The existing system of Indian railway does not support Question Answering system while this novel system will provide anytime anywhere facility of enquiry about railway. The implemented system is quite simple, interactive and easy to use such that end user can ask the query using Hindi transliteration. The results are quite satisfactory as precision and recall values are 98% and 97% respectively. Keywords: Dialogue Manager, Partial Syntactic and Semantic Analysis, Query Processing, Question Answering Railway System

A task manager using an ontological framework for a HARMS-based system

Recently, conversational interaction with technology has made its way into popular commercial use. Advancements in natural language processing have made that possible. Now, imagine a future where the average person can have a team of robots and smart devices working together to accomplish daily tasks and all one has to do is interact with the system naturally. For this theoretical team to exist, an interaction system must be developed that can translate utterances and autonomously organize and direct the agents into completing the task. This paper presents a task manager that uses on ontology to divide tasks into subtasks and finds the most capable agent to complete the task. The task manager is an extension of a robust dialogue manager that users can communication naturally with. Two ontologies were developed as initial steps towards a task manager for a multi-agent system. An experiment was conducted to test the combination of the dialogue manager and the task manager.

Raise the Flag for Mountains: Enhancing Policy Dialogue and Knowledge Sharing through the World Mountain Forum Series

As a mountain country, Switzerland has an intrinsic interest and a proven track record in sustainable mountain development (SMD). Many Swiss stakeholders, including the federal and cantonal administrations, universities, and nongovernmental organizations, actively contribute to global SMD in many ways. Switzerland, with its extensive operational experience in mountainous countries around the world, has been one of the driving forces promoting policy dialogue and knowledge management among different actors to support SMD on various levels. This is reflected in its support for the United Nations’ Agenda 21 and the recent Agenda 2030. Still, after close to 25 years of global policy engagement, the voice of the mountains has not yet gained sufficient momentum and needs further strengthening.