Conversational Interface Research Articles

VerHealth

Healthcare applications on Voice Personal Assistant System (e.g., Amazon Alexa), have shown a great promise to deliver personalized health services via a conversational interface. However, concerns are also raised about privacy, safety, and service quality. In this paper, we propose VerHealth, to systematically assess health-related applications on Alexa for how well they comply with existing privacy and safety policies. VerHealth contains a static module and a dynamic module based on machine learning that can trigger and detect violation behaviors hidden deep in the interaction threads. We use VerHealth to analyze 813 health-related applications on Alexa by sending over 855,000 probing questions and analyzing 863,000 responses. We also consult with three medical school students (domain experts) to confirm and assess the potential violations. We show that violations are quite common, e.g., 86.36% of them miss disclaimers when providing medical information; 30.23% of them store user physical or mental health data without approval. Domain experts believe that the applications' medical suggestions are often factually-correct but are of poor relevance, and applications should have asked more questions before providing suggestions for over half of the cases. Finally, we use our results to discuss possible directions for improvements.

Natural Language Understanding for Multi-Level Distributed Intelligent Virtual Sensors

In this paper we address the problem of automatic sensor composition for servicing human-interpretable high-level tasks. To this end, we introduce multi-level distributed intelligent virtual sensors (multi-level DIVS) as an overlay framework for a given mesh of physical and/or virtual sensors already deployed in the environment. The goal for multi-level DIVS is two-fold: (i) to provide a convenient way for the user to specify high-level sensing tasks; (ii) to construct the computational graph that provides the correct output given a specific sensing task. For (i) we resort to a conversational user interface, which is an intuitive and user-friendly manner in which the user can express the sensing problem, i.e., natural language queries, while for (ii) we propose a deep learning approach that establishes the correspondence between the natural language queries and their virtual sensor representation. Finally, we evaluate and demonstrate the feasibility of our approach in the context of a smart city setup.

Corrigendum to Systematic Evaluation of a Conversational Voice User Interface for Decision-Making

Corrigendum to Systematic Evaluation of a Conversational Voice User Interface for Decision-Making

Can we talk? How a talking agent can improve human autonomy team performance

High levels of automation in future aviation technologies such as Unmanned Aircraft Systems could lead to human operators losing essential Situation Awareness and becoming ‘out-of-the-loop’. Research into Human Autonomy Teaming proposes that improved communication between the human and autonomous agents of a system can address this problem. However, knowledge around the effect of automation audio communication is lacking in the literature and we propose audio-voice conversation would provide the optimum form of communication. In this study we evaluated the impact that providing a conversational interface to a synthetic teammate had on the performance, Situation Awareness and perception of teaming of the human teammate. Twenty-four participants conducted experimental trials on a computer-based task adapted from a Levels Of Automation test method developed by Endsley and Kaber (1999). The results show that synthetic voice communication had a significant positive effect on human performance and perception of teaming. Also demonstrated was that teaming structure has an effect on how that performance increases, with participants in higher Levels Of Automation where the automation provides decision making advice demonstrating a habit of consistently following voice provided advice, even when that advice results in the participant adopting new behaviours and taking more risks.

Benchmarking deep neural network approaches for Indian Sign Language recognition

Sign language is the language of the deaf and mute. However, this particular population of the world is unfortunately overlooked as the majority of the hearing population does not understand sign language. In this paper, an extensive comparative analysis of various gesture recognition techniques involving convolutional neural networks and machine learning algorithms has been discussed and tested for real-time accuracy. Three models: a pre-trained VGG16 with fine-tuning, VGG16 with transfer learning and a hierarchical neural network were analyzed based on a number of trainable parameters. These models were trained on a self-developed dataset consisting images of Indian Sign Language (ISL) representation of all 26 English alphabets. The performance evaluation was based on the practical application of these models, which was simulated by varying lighting and background environments. Out of the three, the hierarchical model outperformed the other two models to give the best accuracy of 98.52% for one-hand and 97% for two-hand gestures. Thereafter, a conversation interface was built in Django using this model for the real-time gesture to speech conversion and vice versa. This publicly accessible interface can be used by anyone who wishes to learn or converse in ISL.

Trainbot: A Conversational Interface to Train Crowd Workers for Delivering On-Demand Therapy

On-demand emotional support is an expensive and elusive societal need that is exacerbated in difficult times — as witnessed during the COVID-19 pandemic. Prior work in affective crowdsourcing has examined ways to overcome technical challenges for providing on-demand emotional support to end users. This can be achieved by training crowd workers to provide thoughtful and engaging on-demand emotional support. Inspired by recent advances in conversational user interface research, we investigate the efficacy of a conversational user interface for training workers to deliver psychological support to users in need. To this end, we conducted a between-subjects experimental study on Prolific, wherein a group of workers (N=200) received training on motivational interviewing via either a conversational interface or a conventional web interface. Our results indicate that training workers in a conversational interface yields both better worker performance and improves their user experience in on-demand stress management tasks.

An empirical study investigating the user acceptance of a virtual conversational agent interface for family health history collection among the geriatric population.

Critical for the early diagnosis of genetic disorders, a Family Health History (FHx) can be collected in several ways including electronic FHx tools, which aid easy editing and sharing by linking with other information management portals. The user acceptance of such systems is critical, especially among older adults experiencing motor and cognitive issues. This study investigated two types of FHx interfaces, standard and Virtual Conversational Agent (VCA), using 30 young (between 18 and 30) and 24 older participants (over 60). Workload, usability and performance data were collected. Even though participants required less time to complete three of five tasks on the standard interface, the VCA interface performed better in terms of subjective workload and usability. Additionally, 67% of the older adults preferred the VCA interface since it provided context-based guidance during the data collection process. The results from this study have implications for the use of virtual assistants in FHx and other areas of data collection.

Improving Access to Justice with Legal Chatbots

On average, one in three Canadians will be affected by a legal problem over a three-year period. Unfortunately, whether it is legal representation or legal advice, the very high cost of these services excludes disadvantaged and most vulnerable people, forcing them to represent themselves. For these people, accessing legal information is therefore critical. In this work, we attempt to tackle this problem by embedding legal data in a conversational interface. We introduce two dialog systems (chatbots) created to provide legal information. The first one, based on data from the Government of Canada, deals with immigration issues, while the second one informs bank employees about legal issues related to their job tasks. Both chatbots rely on various representations and classification algorithms, from mature techniques to novel advances in the field. The chatbot dedicated to immigration issues is shared with the research community as an open resource project.

How to Evaluate Health Applications with Conversational User Interface?

Application of conversational user interfaces (CUI) or chatbots to healthcare is gaining interest fueled by the rising power of artificial intelligence, increasing popularity of mobile health applications and the desire for engagement and usability. While their use is mainly justified by increasing adherence to mobile health applications and facilitating interactions with the system, the question arises: How can such systems be evaluated in a reliable manner? This paper introduces an evaluation framework for health systems whose core interaction principle is a CUI. We derive quality dimensions and attributes by collecting relevant evaluation aspects from applications that have been developed in previous work and from literature on health chatbots. The collected aspects are aggregated into six thematic categories for chatbot quality, including user experience, linguistic, task-oriented and artificial intelligence perspectives, but also healthcare quality and system quality perspectives. The framework is intended to support developers and researchers in the domain of chatbots in healthcare in selecting relevant quality attributes to be assessed before their systems are distributed to patients.

Clinical Screening Interview Using a Social Robot for Geriatric Care

Social robots are coming to our homes and have already been used to help humans in a number of ways in geriatric care. This article aims to develop a framework that enables social robots to conduct regular clinical screening interviews in geriatric care, such as cognitive evaluation, falls’ risk evaluation, and pain rating. We develop a social robot with essential features to enable clinical screening interviews, including a conversational interface, face tracking, an interaction handler, attention management, robot skills, and cloud service management. Besides, a general clinical screening interview management (GCSIM) model is proposed and implemented. The GCSIM enables social robots to handle various types of clinical questions and answers, evaluate and score responses, engage interviewees during conversations, and generate reports on their well-being. These reports can be used to evaluate the progression of cognitive impairment, risk of falls, pain level, and so on by caregivers or physicians. Such a clinical screening capability allows for early detection and treatment planning in geriatric care. The framework was developed and implemented on our 3-D-printed social robot. It was tested on 30 older adults with different ages, achieved satisfying results, and received their high confidence and trust in the use of this robot for human well-being assessment. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article is motivated by the goal of using a social robot to perform geriatric well-being assessment through clinical screening interviews. In order to conduct clinical screening interviews, the social robot needs the following essential features: having a verbal conversational interface, adapting to different types of clinical screening interviews, scoring and evaluating answers, having nondirective listening responses, and enabling directive listening responses. The proposed general clinical screening interview management (GCSIM) model demonstrates these capabilities on the social robot. The robot can give structured clinical screening interviews with different question–answer sheets. This will help advance assistive technologies for use by geriatric physicians, nurses, and social service professionals to keep older adults healthy, safe, and independent at home. Robots will become more and more essential in working alongside geriatric practitioners to help monitor older adults at home and to provide early detection and warning of cognitive/mental health problems, falls’ risk, and so on. This early detection property can improve quality-of-care and help older adults remain living at home.

Estimating Conversational Styles in Conversational Microtask Crowdsourcing

Crowdsourcing marketplaces have provided a large number of opportunities for online workers to earn a living. To improve satisfaction and engagement of such workers, who are vital for the sustainability of the marketplaces, recent works have used conversational interfaces to support the execution of a variety of crowdsourcing tasks. The rationale behind using conversational interfaces stems from the potential engagement that conversation can stimulate. Prior works in psychology have also shown that conversational styles can play an important role in communication. There are unexplored opportunities to estimate a worker's conversational style with an end goal of improving worker satisfaction, engagement and quality. Addressing this knowledge gap, we investigate the role of conversational styles in conversational microtask crowdsourcing. To this end, we design a conversational interface which supports task execution, and we propose methods to estimate the conversational style of a worker. Our experimental setup was designed to empirically observe how conversational styles of workers relate with quality-related outcomes. Results show that even a naive supervised classifier can predict the conversation style with high accuracy (80%), and crowd workers with an Involvement conversational style provided a significantly higher output quality, exhibited a higher user engagement and perceived less cognitive task load in comparison to their counterparts. Our findings have important implications on task design with respect to improving worker performance and their engagement in microtask crowdsourcing.

Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset

Virtual assistants such as Google Assistant, Alexa and Siri provide a conversational interface to a large number of services and APIs spanning multiple domains. Such systems need to support an ever-increasing number of services with possibly overlapping functionality. Furthermore, some of these services have little to no training data available. Existing public datasets for task-oriented dialogue do not sufficiently capture these challenges since they cover few domains and assume a single static ontology per domain. In this work, we introduce the the Schema-Guided Dialogue (SGD) dataset, containing over 16k multi-domain conversations spanning 16 domains. Our dataset exceeds the existing task-oriented dialogue corpora in scale, while also highlighting the challenges associated with building large-scale virtual assistants. It provides a challenging testbed for a number of tasks including language understanding, slot filling, dialogue state tracking and response generation. Along the same lines, we present a schema-guided paradigm for task-oriented dialogue, in which predictions are made over a dynamic set of intents and slots, provided as input, using their natural language descriptions. This allows a single dialogue system to easily support a large number of services and facilitates simple integration of new services without requiring additional training data. Building upon the proposed paradigm, we release a model for dialogue state tracking capable of zero-shot generalization to new APIs, while remaining competitive in the regular setting.

Submitting surveys via a conversational interface: An evaluation of user acceptance and approach effectiveness

Conversational interfaces are currently on the rise: more and more applications rely on a chat-like interaction pattern to increase their acceptability and to improve user experience. Also in the area of questionnaire design and administration, interaction design is increasingly looked at as an important ingredient of a digital solution. For those reasons, we designed and developed a conversational survey tool to administer questionnaires with a colloquial form through a chat-like Web interface. In this paper, we present the evaluation results of our approach, taking into account both the user point of view - by assessing user acceptance and preferences in terms of survey compilation experience - and the survey design perspective - by investigating the effectiveness of a conversational survey in comparison to a traditional questionnaire. We show that users clearly appreciate the conversational form and prefer it over a traditional approach and that, from a data collection point of view, the conversational method shows the same reliability and a higher response quality with respect to a traditional questionnaire.

Understanding Physician’s Experience With Conversational Interfaces During Occupational Health Consultation

This paper presents an exploratory study on using conversational interfaces (CIs) to support physicians in conducting occupational health consultations. The CI was achieved through a web-based information dashboard with a chatbot assistant for providing real-time suggestions through text messages. Two system designs were developed: the first using a proactive chatbot, the second using an on-demand type of interaction. The effectiveness of the proposed CI and the two types of chatbot designs were investigated in a field study consisted of eight healthcare consultations. Quantitative results showed that the CI was positively evaluated as a reliable tool to be used during medical consultations and that occupational health physicians were eager to use this technology in their work. The qualitative data analysis suggested that our design concept might improve the workflow during the consultation, particularly with respect to the access to relevant information and structured decision-making processes using valuable references. The on-demand, lightweight type of chatbot interaction was better perceived than the proactive one. Based on these findings, we discuss implications for the future development of occupational health consultation based on CI and their potential contribution to computer-assisted, data-driven healthcare.

Computational Psycholinguistics

Computational Psycholinguistics

Perceiving a Mind in a Chatbot: Effect of Mind Perception and Social Cues on Co-presence, Closeness, and Intention to Use

ABSTRACTA chatbot equipped with a conversational user interface often allows its users to feel as if they are conversing with a human being. The current study examined whether users’ perception of a mind within a chatbot is associated with their feeling of co-presence, closeness, and intention to use and whether the influence of mind perception is reinforced when the chatbot presents social cues in its language. A laboratory experiment (N = 64) revealed that the more participants perceived a mind behind a chatbot, the more co-presence and interpersonal closeness they experienced with the chatbot. The associations with co-presence and closeness became stronger when the chatbot used social cues. Furthermore, mind perception had an indirect effect on intention to use via closeness when social cues were presented. These findings imply the importance of mind perception and social cues in a chatbot’s language in creating a positive chatbot experience.

Conversational Interfaces for Health: Bibliometric Analysis of Grants, Publications, and Patents.

BackgroundConversational interfaces (CIs) in different modalities have been developed for health purposes, such as health behavioral intervention, patient self-management, and clinical decision support. Despite growing research evidence supporting CIs’ potential, CI-related research is still in its infancy. There is a lack of systematic investigation that goes beyond publication review and presents the state of the art from perspectives of funding agencies, academia, and industry by incorporating CI-related public funding and patent activities.ObjectiveThis study aimed to use data systematically extracted from multiple sources (ie, grant, publication, and patent databases) to investigate the development, research, and fund application of health-related CIs and associated stakeholders (ie, countries, organizations, and collaborators).MethodsA multifaceted search query was executed to retrieve records from 9 databases. Bibliometric analysis, social network analysis, and term co-occurrence analysis were conducted on the screened records.ResultsThis review included 42 funded projects, 428 research publications, and 162 patents. The total dollar amount of grants awarded was US $30,297,932, of which US $13,513,473 was awarded by US funding agencies and US $16,784,459 was funded by the Europe Commission. The top 3 funding agencies in the United States were the National Science Foundation, National Institutes of Health, and Agency for Healthcare Research and Quality. Boston Medical Center was awarded the largest combined grant size (US $2,246,437) for 4 projects. The authors of the publications were from 58 countries and 566 organizations; the top 3 most productive organizations were Northeastern University (United States), Universiti Teknologi MARA (Malaysia), and the French National Center for Scientific Research (CNRS; France). US researchers produced 114 publications. Although 82.0% (464/566) of the organizations engaged in interorganizational collaboration, 2 organizational research-collaboration clusters were observed with Northeastern University and CNRS as the central nodes. About 112 organizations from the United States and China filed 87.7% patents. IBM filed most patents (N=17). Only 5 patents were co-owned by different organizations, and there was no across-country collaboration on patenting activity. The terms patient, child, elderly, and robot were frequently discussed in the 3 record types. The terms related to mental and chronic issues were discussed mainly in grants and publications. The terms regarding multimodal interactions were widely mentioned as users’ communication modes with CIs in the identified records.ConclusionsOur findings provided an overview of the countries, organizations, and topic terms in funded projects, as well as the authorship, collaboration, content, and related information of research publications and patents. There is a lack of broad cross-sector partnerships among grant agencies, academia, and industry, particularly in the United States. Our results suggest a need to improve collaboration among public and private sectors and health care organizations in research and patent activities.

Data Science Recommendation System using Semantic Technology

Data Science is a field in which multidisciplinary blend of data inference, algorithm development, and technology are combined in order to solve analytically complex problems. Data science field tend to focus on a complex and large algorithmic technological problem having data at the core. Students at the starting of the learning phase don’t know all the technological and algorithmic aspects related to data science. Consulting through faculty or knowing from external source helps them to proceed towards the needed expertise that they want to gain. Data science recommendation system using semantic web data-science ontology and service-oriented architecture is proposed in our work to recommend students the appropriate resources for their queries. Recommending student significant information concerning books, on-line documentation, software tools, public code repositories, and experts, tutors to be contacted based on student query or previous exam scores is the main objective of the work. Ontology based service-oriented web platform with a conversational user interface that use NLP (Natural Language Processing) to recommend possible resources is proposed and tested to be efficient

Assessing long-term user experience on a mobile health application through an in-app embedded conversation-based questionnaire

A satisfying experience is critical for the user to feel motivated over time, especially in mobile health (m-Health) applications. Long-term user experience (UX) assesses more extended periods of use, which may reveal the possible impact of a set of experiences. In this study, using an in-app embed questionnaire available through a conversational interface, we investigated long-term UX on an m-Health application to identify how it occurs over time. The methods were developed in 4 phases, which included problem characterization, search for UX aspects, elaboration and verification of the questionnaire, test period with questionnaire applied before, during and after use, interview with users, and information mapping. For 3 months, 37 users reported their experiences with the application, describing their impressions regarding usage. Results demonstrated that the most satisfying experiences occurred mainly within the first weeks of the study and were associated with the app's features, charts and visual resources, and the practicality of treatment monitoring. For the less satisfactory experiences, the main reasons identified were technical problems on the application, and the effort and difficulty of use. All users appreciated the use of a character in the conversational interface as a gatherer of the answers to the assessment questionnaire.

InstructableCrowd: Creating IF-THEN Rules for Smartphones via Conversations with the Crowd

Natural language interfaces have become a common part of modern digital life. Chatbots utilize text-based conversations to communicate with users; personal assistants on smartphones such as Google Assistant take direct speech commands from their users; and speech-controlled devices such as Amazon Echo use voice as their only input mode. In this paper, we introduce InstructableCrowd, a crowd-powered system that allows users to program their devices via conversation. The user verbally expresses a problem to the system, in which a group of crowd workers collectively respond and program relevant multi-part IF-THEN rules to help the user. The IF-THEN rules generated by InstructableCrowd connect relevant sensor combinations (e.g., location, weather, device acceleration, etc.) to useful effectors (e.g., text messages, device alarms, etc.). Our study showed that non-programmers can use the conversational interface of InstructableCrowd to create IF-THEN rules that have similar quality compared with the rules created manually. InstructableCrowd generally illustrates how users may converse with their devices, not only to trigger simple voice commands, but also to personalize their increasingly powerful and complicated devices.