Automatic Count of Bites and Chews From Videos of Eating Episodes.

Delwar Hossain,Tonmoy Ghosh,Edward Sazonov

doi:10.1109/access.2020.2998716

Delwar Hossain, Tonmoy Ghosh + Show 1 more

Open Access

https://doi.org/10.1109/access.2020.2998716

Copy DOI

Abstract

Methods for measuring of eating behavior (known as meal microstructure) often rely on manual annotation of bites, chews, and swallows on meal videos or wearable sensor signals. The manual annotation may be time consuming and erroneous, while wearable sensors may not capture every aspect of eating (e.g. chews only). The aim of this study is to develop a method to detect and count bites and chews automatically from meal videos. The method was developed on a dataset of 28 volunteers consuming unrestricted meals in the laboratory under video observation. First, the faces in the video (regions of interest, ROI) were detected using Faster R-CNN. Second, a pre-trained AlexNet was trained on the detected faces to classify images as a bite/no bite image. Third, the affine optical flow was applied in consecutively detected faces to find the rotational movement of the pixels in the ROIs. The number of chews in a meal video was counted by converting the 2-D images to a 1-D optical flow parameter and finding peaks. The developed bite and chew count algorithm was applied to 84 meal videos collected from 28 volunteers. A mean accuracy (±STD) of 85.4% (±6.3%) with respect to manual annotation was obtained for the number of bites and 88.9% (±7.4%) for the number of chews. The proposed method for an automatic bite and chew counting shows promising results that can be used as an alternative solution to manual annotation.

Highlights

Study of eating behavior of individuals is important for researchers to understand the eating patterns of people suffering from obesity and eating disorders
We propose a novel contactless bite and chew counting method from video based on object detection, image classification, and optical flow
The trained face detector achieved an average IoU of 0.97±0.01 and an average mAP

Summary

Introduction

Study of eating behavior of individuals is important for researchers to understand the eating patterns of people suffering from obesity and eating disorders. Meal microstructure combines factors of food intake behavior such as total eating episode duration (start of the food intake to the end including pauses), true ingestion duration (time spent in chewing), eating event number (a bite which is potentially followed by a segment of chewing and swallowing), ingestion rate, frequency and efficiency of chewing, and size of bite [4]. In [6], the authors claimed that eating rate feedback can be helpful to aid in intervention in eating disorder treatment. Health researchers are applying the number of chews and the rate of chewing as variables in their models for estimation of ingested mass and energy intake. Counts of chews and swallows were used to develop the energy intake estimation model (CCS) in [10]. In [11] authors concluded that for solid food, recordings of chewing sound can be used to predict bite weight

Objectives

Methods

Results

Discussion

Conclusion