PastVision+: Thermovisual Inference of Recent Medicine Intake by Detecting Heated Objects and Cooled Lips

Martin Cooney,Josef Bigun

doi:10.3389/frobt.2017.00061

Abstract

This article addresses the problem of how a robot can infer what a person has done recently, with a focus on checking oral medicine intake in dementia patients. We present PastVision+, an approach showing how thermovisual cues in objects and humans can be leveraged to infer recent unobserved human-object interactions. Our expectation is that this approach can provide enhanced speed and robustness compared to existing methods, because our approach can draw inferences from single images without needing to wait to observe ongoing actions and can deal with short-lasting occlusions; when combined, we expect a potential improvement in accuracy due to the extra information from knowing what a person has recently done. To evaluate our approach, we obtained some data in which an experimenter touched medicine packages and a glass of water to simulate intake of oral medicine, for a challenging scenario in which some touches were conducted in front of a warm background. Results were promising, with a detection accuracy of touched objects of 50\% at the 15 seconds mark and 0\% at the 60 seconds mark, and a detection accuracy of cooled lips of about 100\% and 60\% at the 15 seconds mark for cold and tepid water respectively. Furthermore, we conducted a follow-up check for another challenging scenario in which some participants pretended to take medicine or otherwise touched a medicine package: accuracies of inferring object touches, mouth touches, and actions were 72.2\%, 80.3\%, and 58.3\% initially, and 50.0\%, 81.7\%, and 50.0\% at the 15 seconds mark, with a rate of 89.0\% for person identification. The results suggested some areas in which further improvements would be possible, toward facilitating robot inference of human actions, in the context of medicine intake monitoring.

Highlights

This article addresses the problem of how a robot can detect what a person has touched recently, with a focus on checking oral medicine intake in dementia patients.Detecting recent touches would be useful because touch is a typical component of many human– object interactions; knowing which objects have been touched allows inference into what actions have been conducted, which is an important requirement for robots to collaborate effectively with people (Vernon et al, 2016)
PastVision+ allows quick inference from single images, which can be processed in several seconds, and requires only inexpensive components: a small thermal camera, costing approximately two hundred US dollars, a regular camera, and a computer
We observed that our approach could be used in a challenging scenario in which some touches occurred in regions where the background was warm, with about 50% detection accuracy of touched objects and 100% detection accuracy of cooled lips 15 s after a touch

Summary

Introduction

This article addresses the problem of how a robot can detect what a person has touched recently, with a focus on checking oral medicine intake in dementia patients.Detecting recent touches would be useful because touch is a typical component of many human– object interactions; knowing which objects have been touched allows inference into what actions have been conducted, which is an important requirement for robots to collaborate effectively with people (Vernon et al, 2016). PastVision+: Thermovisually Inferring Medicine Intake importance has been described in the literature (Osterberg and Blaschke, 2005) and which can be problematic for dementia patients who might not remember to take medicine—and in particular on oral medicination, which is a common administration route. Within this context, it is not always possible for a robot to observe people due to occlusions and other tasks the robot might be expected to do; the capability to detect what a person has recently touched, from a few seconds to a few minutes ago, would be helpful. A challenge was that it was unclear how an algorithm could be designed to detect such touches in a typical scenario in which both foreground and background can comprise regions of similar temperature, for several seconds after contact had occurred

Methods

Results

Conclusion