Topic modelling for routine discovery from egocentric photo-streams

Estefania Talavera,Carolin Wuerich,Nicolai Petkov,Petia Radeva

doi:10.1016/j.patcog.2020.107330

Estefania Talavera, Carolin Wuerich + Show 2 more

Open Access

https://doi.org/10.1016/j.patcog.2020.107330

Copy DOI

Abstract

Developing tools to understand and visualize lifestyle is of high interest when addressing the improvement of habits and well-being of people. Routine, defined as the usual things that a person does daily, helps describe the individuals’ lifestyle. With this paper, we are the first ones to address the development of novel tools for automatic discovery of routine days of an individual from his/her egocentric images. In the proposed model, sequences of images are firstly characterized by semantic labels detected by pre-trained CNNs. Then, these features are organized in temporal-semantic documents to later be embedded into a topic models space. Finally, Dynamic-Time-Warping and Spectral-Clustering methods are used for final day routine/non-routine discrimination. Moreover, we introduce a new EgoRoutine-dataset, a collection of 104 egocentric days with more than 100.000 images recorded by 7 users. Results show that routine can be discovered and behavioural patterns can be observed.

Highlights

With the dynamization of the day-by-day of our century, many people need to improve the quality of their life, and the first step is to get a better understanding of it
Our goal is to address the routine discovery by analyzing the appearance of these patterns in the life of a person
We present quantitative and qualitative results of the performance on the different stages of our approach for routine discovery validated on our EgoRoutine dataset

Summary

Introduction

With the dynamization of the day-by-day of our century, many people need to improve the quality of their life, and the first step is to get a better understanding of it. In [29], the authors claimed that the definition of patterns of behaviour allows people to reach goals by creating associations between actions that are repeated in a stable context and their responses. We relate patterns to the combination of elements that describe the context of the days of someone, such as: environment, objects around the person, and his/her activities. Patterns of behaviour were described as ordered sequences of activities [16] and are important elements when describing the Routine of a person. Routine describes habits and sequences of activities of someone’s days, and tends to be unique. The ability to perform Activities of Daily Living (ADL) directly affects someone’s quality of life. The discovery of the routine of a person is of importance for its later analysis in order to assure the healthy living of individuals

Objectives

Methods

Results

Conclusion