Responding to subtle, fleeting changes in the user's internal state

Abstract

In human-to-human interaction, people sometimes are able to pick up and respond sensitively to the other's internal state as it shifts moment by moment over the course of an exchange. To find out whether such an ability is worthwhile for computer human interfaces, we built a semi-automated tutoring-type spoken dialog system. The system inferred information about the user's \scare{ephemeral emotions}, such as confidence, confusion, pleasure, and dependency, from the prosody of his utterances and the context. It used this information to select the most appropriate acknowledgement form at each moment. In doing so the system was following some of the basic social conventions for real-time interaction. Users rated the system with this ability more highly than a version without.