Human Wizard Research Articles

Designing the Conversational Agent: Asking Follow-up Questions for Information Elicitation

Conversational Agents (CAs) can facilitate information elicitation in various scenarios, such as semi-structured interviews. Current CAs can ask predetermined questions but lack skills for asking follow-up questions. Thus, we designed three approaches for CAs to automatically ask follow-up questions, i.e., follow-ups on concepts, follow-ups on related concepts, and general follow-ups. To investigate their effects, we conducted a user study (N=26) in which a CA interviewer asked follow-up questions generated by algorithms and crafted by human wizards. Our results showed that the CA's follow-up questions were readable and effective in information elicitation. The follow-ups on concepts and related concepts achieved a lower drop rate and better relevance, while the general follow-ups elicited more informative responses. Further qualitative analysis of the human-CA interview data revealed algorithm drawbacks and identified follow-up question techniques used by the human wizards. We provided design implications for improving information elicitation of future CAs based on the results.

Coherent visual design through attribute-specific feedback: a hybrid approach to intelligent design agents

ABSTRACT The scope of visual design is expanding to promote products and services across digital communication platforms, but support for coherent design with style guides seems limited. Based on the literature on style, coherence and design support tools with different levels of intelligence, we describe coherence in visual design and propose to integrate preset style guidelines and constraints into feedback interactions to support achieving visual coherence. After formative research with expert designers, we prototype a pseudo-AI design support tool to simulate attribute-specific feedback and conduct user research to probe how participants engage with feedback provided by a human wizard acting as an intelligent agent. We analyze patterns of feedback and types of reactions observed in participants’ design processes and outcomes. We then discuss the implications of attribute-specific feedback to describe visual coherence as a meta-property of design, and intelligent agents that guide situational judgements about visual coherence through feedback interactions.

Towards a Tutoring System to Support Robotics Activities in Classrooms – Two Wizard-of-Oz Studies

This paper develops a method for the construction and evaluation of cognitive models to support students in their problem-solving skills during robotics in school, aiming to build a basis for an implementation of a tutoring system in the future. Two Wizard-of-Oz studies were conducted, one in the classroom and one in the lab. Based on the cognitive model, the human wizards gave support to 20 students working in pairs. The studies were video recorded and a qualitative analysis was conducted. This qualitative research approach is described in detail. The evaluation of the studies showed that students reacted mostly positively to the wizards. We also uncovered ways in which students’ problem-solving skills could be improved. Based on the evaluation and observations of the Wizard-of-Oz studies, the paper proposes a design for a future robotics skills tutoring system.

How Service Robots Can Improve Workplace Experience: Camaraderie, Customization, and Humans-in-the-Loop.

This paper presents the results of a three-week in-the-wild deployment of a wizarded service robot in a shared campus workplace. The study introduces robot-centric ethnography, a concept in which a wizarded robot acts as a mediated anthropologist, used in this case, to further our understandings of how service robots impact and integrate into everyday workplace experiences. Our research site included participants familiar with robots, recruited from 90+ students and faculty working in a shared lab space. Our wizarding team visited these participants each workday they were there for three weeks, navigating open office and lab spaces to remind participants to be aware of their mental, physical, and nutritional health needs. Using a semi-structured format, the wizards adapted the standard interaction flow to the situation. This interaction sequence was guided via pre-populated buttons on our health coach interface, with human wizards triggering the timing and adding extra responses as felt natural. Our ethnography-informed approach used the social knowledge of both participants and wizards, blending the robot into the cultural environment in which it was operating. Our data supports the positive impact of fluent service robot experience on participant mood and overall workplace experience. This suggests that effectively designed service robots can benefit workplace environments above and beyond their intended functions.

Is a Wizard-of-Oz Required for Robot-Led Conversation Practice in a Second Language?

The large majority of previous work on human-robot conversations in a second language has been performed with a human wizard-of-Oz. The reasons are that automatic speech recognition of non-native conversational speech is considered to be unreliable and that the dialogue management task of selecting robot utterances that are adequate at a given turn is complex in social conversations. This study therefore investigates if robot-led conversation practice in a second language with pairs of adult learners could potentially be managed by an autonomous robot. We first investigate how correct and understandable transcriptions of second language learner utterances are when made by a state-of-the-art speech recogniser. We find both a relatively high word error rate (41%) and that a substantial share (42%) of the utterances are judged to be incomprehensible or only partially understandable by a human reader. We then evaluate how adequate the robot utterance selection is, when performed manually based on the speech recognition transcriptions or autonomously using (a) predefined sequences of robot utterances, (b) a general state-of-the-art language model that selects utterances based on learner input or the preceding robot utterance, or (c) a custom-made statistical method that is trained on observations of the wizard’s choices in previous conversations. It is shown that adequate or at least acceptable robot utterances are selected by the human wizard in most cases (96%), even though the ASR transcriptions have a high word error rate. Further, the custom-made statistical method performs as well as manual selection of robot utterances based on ASR transcriptions. It was also found that the interaction strategy that the robot employed, which differed regarding how much the robot maintained the initiative in the conversation and if the focus of the conversation was on the robot or the learners, had marginal effects on the word error rate and understandability of the transcriptions but larger effects on the adequateness of the utterance selection. Autonomous robot-led conversations may hence work better with some robot interaction strategies.

Are you convinced? A Wizard of Oz study to test emotional vs. rational persuasion strategies in dialogues

This research explores ideal methods of persuasion through computer-mediated dialogue. We attempt to identify which persuasive strategy is most successful. We designed a Wizard of Oz laboratory experiment, where participants interact with a human wizard via a custom-developed web-based chat interface. The wizard attempted to persuade participants to learn more about Tai Chi using the following persuasive strategies: Emotional Positive, Emotional Negative, Rational Positive, and Rational Negative. Based on the results of the pre- and post-chat questionnaire, participants’ interest in learning Tai Chi was significantly greater after completing the dialogue and 69% percent of the participants printed a flyer to receive more information. Furthermore, conversations using the Emotional Positive strategies resulted in more successful persuasion than rational ones. The results of our study suggest that Emotional Positive strategies may be the most effective. We also suggest successful strategies as a design guideline for autonomous dialogue systems for persuasion.

A Supervised Learning Framework for Modeling Director Agent Strategies in Educational Interactive Narrative

Computational models of interactive narrative offer significant potential for creating educational game experiences that are procedurally tailored to individual players and support learning. A key challenge posed by interactive narrative is devising effective director agent models that dynamically sequence story events according to players' actions and needs. In this paper, we describe a supervised machine-learning framework to model director agent strategies in an educational interactive narrative Crystal Island. Findings from two studies with human participants are reported. The first study utilized a Wizard-of-Oz paradigm where human “wizards” directed participants through Crystal Island's mystery storyline by dynamically controlling narrative events in the game environment. Interaction logs yielded training data for machine learning the conditional probabilities of a dynamic Bayesian network (DBN) model of the human wizards' directorial actions. Results indicate that the DBN model achieved significantly higher precision and recall than naive Bayes and bigram model techniques. In the second study, the DBN director agent model was incorporated into the runtime version of Crystal Island, and its impact on students' narrative-centered learning experiences was investigated. Results indicate that machine-learning director agent strategies from human demonstrations yield models that positively shape players' narrative-centered learning and problem-solving experiences.

Naturalistic Dialogue Management for Noisy Speech Recognition

With naturalistic dialogue management, a spoken dialogue system behaves as a human would under similar conditions. This paper reports on an experiment to develop naturalistic clarification strategies for noisy speech recognition in the context of spoken dialogue systems. We collected a wizard-of-Oz corpus in which human wizards with access to a rich set of clarification actions made clarification decisions online, based on human-readable versions of system data. The experiment compares an evaluation of calls to a baseline system in a library domain with calls to an enhanced version of the system. The new system has a clarification module based on the wizard data that is a decision tree constructed from three machine-learned models. It replicates the wizards' ability to ground partial understandings of noisy input and to build upon them. The enhanced system has a significantly higher rate of task completion, greater task success and improved efficiency.

Towards a navigation system for blind people

In this paper, we present an initial study towards of an indoor navigation system for blind people. As the system itself is still in an early stage of development, we conducted a Wizard of Oz study using a generic Wizard of Oz system designed for mobile and ubiquitous studies. The goal of the study was to validate a set of audio-based navigation commands in a field study context. Further, we wanted to identify usability issues of the Wizard of Oz tool, and ensure the appropriateness of the addressed study setup. Therefore, we used eight human wizards as participants in the study. Their task was to guide two blindfolded actors through a predefined route. Such settings helped us to achieve high ecological validity of the results compared to laboratory testing. We found that the developed study setup is fully mobile and can be used in any mobile context, the voice commands chosen for navigation are almost complete, and can be used with slight modifications for the follow-up study. Additionally we identified several usability flaws of the Wizard of Oz tool. After implementing the findings, the tool and the study setup are ready for a follow-up study with blind persons in order to validate the selected voice commands in depth.

Learning and Evaluation of Dialogue Strategies for New Applications: Empirical Methods for Optimization from Small Data Sets

We present a new data-driven methodology for simulation-based dialogue strategy learning, which allows us to address several problems in the field of automatic optimization of dialogue strategies: learning effective dialogue strategies when no initial data or system exists, and determining a data-driven reward function. In addition, we evaluate the result with real users, and explore how results transfer between simulated and real interactions. We use Reinforcement Learning (RL) to learn multimodal dialogue strategies by interaction with a simulated environment which is “bootstrapped” from small amounts of Wizard-of-Oz (WOZ) data. This use of WOZ data allows data-driven development of optimal strategies for domains where no working prototype is available. Using simulation-based RL allows us to find optimal policies which are not (necessarily) present in the original data. Our results show that simulation-based RL significantly outperforms the average (human wizard) strategy as learned from the data by using Supervised Learning. The bootstrapped RL-based policy gains on average 50 times more reward when tested in simulation, and almost 18 times more reward when interacting with real users. Users also subjectively rate the RL-based policy on average 10% higher. We also show that results from simulated interaction do transfer to interaction with real users, and we explicitly evaluate the stability of the data-driven reward function.

Learning human multimodal dialogue strategies

AbstractWe investigate the use of different machine learning methods in combination with feature selection techniques to explore human multimodal dialogue strategies and the use of those strategies for automated dialogue systems. We learn policies from data collected in a Wizard-of-Oz study where different human ‘wizards’ decide whether to ask a clarification request in a multimodal manner or else to use speech alone. We first describe the data collection, the coding scheme and annotated corpus, and the validation of the multimodal annotations. We then show that there is a uniform multimodal dialogue strategy across wizards, which is based on multiple features in the dialogue context. These are generic features, available at runtime, which can be implemented in dialogue systems. Our prediction models (for human wizard behaviour) achieve a weighted f-score of 88.6 per cent (which is a 25.6 per cent improvement over the majority baseline). We interpret and discuss the learned strategy. We conclude that human wizard behaviour is not optimal for automatic dialogue systems, and argue for the use of automatic optimization methods, such as Reinforcement Learning. Throughout the investigation we also discuss the issues arising from using small initial Wizard-of-Oz data sets, and we show that feature engineering is an essential step when learning dialogue strategies from such limited data.

The Igbe Cult in Isokoland and Missionary and Government Reactions, 1915-1930

It is my aim in this paper to account for the dramatic historical emergence of the neo-primal syncretistic Igbe cult in Isokoland of Nigeria in the light of the socio-religious circumstances of the period preceding the rise of the cult. I wish to bring into focus the real socio-religious dimension of the cult, contrary to what seems to be a misunderstanding on the part of the alien crusaders who became its attackers. On the basis of the fundamental beliefs and organization of the cult underlying this rise, I wish to ascertain whether or not the movement had anything to do with the conflict arising from culture contact or from the people's reaction against foreign invaders, as is certainly the case with such movements in some preliterate societies. Our findings will enable us to conclude whether the opposition launched against it by the missionaries and the colonial regime was justifiable or not. (a) Witchcraft beliefs and practices in the territory The problem of witchcraft beliefs and practices and other allied antisocial evils appears to be the root cause of the rise of the Igbe cult in Isoko territory. A brief look at the basic Isoko beliefs in witchcraft and their early mode of control of its practice becomes necessary here. ^ The Isoko believe that there are spirit powers and principalities that may be essentially malignant and that these may have the power of operating through human agents, particularly witches and wizards (erieda). Idowu has rightly described these as human beings of very strong wills and with diabolical bent; veritably wicked ones who derive sadistic satisfaction from bringing misfortune upon

The Withdrawal of the Fertility God

ONE of the best-known tales in Norse mythology is the story of Skirnir's journey to woo Gerd, the sea-giant Gymir's daughter, on behalf of his master, the Vanr-god Frey. It is told in two slightly different versions: Skirnismdl, a poem in the poetic Edda, also called F6r Skirnis, and a prose version by Snorri Sturluson, the 13th century Icelandic poet and historian.' Snorri's version is found in Gylfaginning, part II of his Edda, the so-called prose Edda which he wrote about 1223, three centuries or so, it is thought, after the composition of Skirnismndl, but there are no definite pointers to the date of the poem.2 Gylfaginning, 'The Deception of Gylfi,' is a frame story about the visit of a Swedish king to the abode of the Aesir, who in Snorri's narrative are not gods but human wizards. For Skirnis is one of the mythic tales contained in this frame. In a few introductory lines to Skirnismndl we are told that one day Njord's son Frey sat down in the seat called Hlidscidlf From there he could overlook the whole world including Jditunheim, the home of the giants, where he saw a beautiful young woman walking across the yard in Gymir's homestead. This sight caused him much mental anguish, and Njord and his wife Skadi summoned Skirnir, Frey's trusted servant, and asked him to find out what was troubling his master. In the poem Skirnir does what is requested, but with some reluctance since he expects angry words from his master if he disturbs him. After some persuasion Frey reveals to him the reason for his withdrawal: he happened to see a maiden in Gymir's stronghold and fell in love with her, but sadly doubts that the gods and elves will allow the consummation of this love for Gymir's daughter. Skirnir offers to woo the maiden on his behalf on condition that Frey gives him a horse that can carry him through the dark magic curtain of fire surrounding Gymir's stronghold, and the sword that fights by itself against the giants. This is granted him at once, and in a single stanza spoken by Skirnir to the horse we are told about the dangers they are encountering on the journey. It is made in darkness across the wet mountains in giant-land, where they may be caught by the loathsome giant. A brief narrative in prose informs us that Skirnir has arrived at Gymir's stronghold, where he faces a giant shepherd sitting on a mound by the gate where fierce hounds are tethered. The poem resumes with a dialogue between Skirnir and the shepherd, who tells him he will never be able to pass the gate alive or speak to Gymir's daughter; but Skirnir, who believes that whatever action he takes now cannot alter the day on which he is doomed to die, fights his way past the shepherd into the stronghold. He is received by Gerd, who with her servant has heard the din of the fight and guesses that the dismounting stranger must have killed her brother, presumably the guardian shepherd. When she learns the reason for his visit,