Abstract

Theory of mind, i.e., the ability to reason about intents and beliefs of agents is an important task in artificial intelligence and central to resolving ambiguous references in natural language dialogue. In this work, we revisit the evaluation of theory of mind through question answering. We show that current evaluation methods are flawed and that existing benchmark tasks can be solved without theory of mind due to dataset biases. Based on prior work, we propose an improved evaluation protocol and dataset in which we explicitly control for data regularities via a careful examination of the answer space. We show that state-of-the-art methods which are successful on existing benchmarks fail to solve theory-of-mind tasks in our proposed approach.

Highlights

  • Humans interact and communicate with other people in a highly efficient way, as described for instance as Grice’s cooperative principle (Grice et al, 1975)

  • The first question tests the ability of the child to infer the correct mental state of Sally, i.e., that she has the false belief of the marble being in the basket

  • Theory of mind is an important component of intelligent systems which interact with humans

Read more

Summary

Introduction

Humans interact and communicate with other people in a highly efficient way, as described for instance as Grice’s cooperative principle (Grice et al, 1975). The key insight of Grant et al (2017) was to cast them as question answering tasks, where a system is given a story and has to answer questions about the beliefs of agents in that story This allows to adapt the bAbi benchmarking protocol (Weston et al, 2016) to evaluate theory of mind capabilities of modern neural network architectures: stories are automatically generated so that a suitably large number of examples can be provided for training. We show that stateof-the-art memory-augmented models – which are successful on existing benchmarks – fail to solve theory-of-mind tasks in our improved approach

Theory of Mind Benchmarks
Evaluating Theory of Mind Evaluation
Experimental Evaluation
Findings
Discussion
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.