Deep Learning for Intake Gesture Detection From Wrist-Worn Inertial Sensors: The Effects of Data Preprocessing, Sensor Modalities, and Sensor Positions

Hamid Heydarian,Clare E Collins,Marc T P Adam,Tracy Burrows,Megan E Rollo,Philipp V Rouast

doi:10.1109/access.2020.3022042

Hamid Heydarian, Clare E Collins + Show 4 more

Open Access

https://doi.org/10.1109/access.2020.3022042

Copy DOI

Abstract

Wrist-worn inertial measurement units have emerged as a promising technology to passively capture dietary intake data. State-of-the-art approaches use deep neural networks to process the collected inertial data and detect characteristic hand movements associated with intake gestures. In order to clarify the effects of data preprocessing, sensor modalities, and sensor positions, we collected and labeled inertial data from wrist-worn accelerometers and gyroscopes on both hands of 100 participants in a semi-controlled setting. The method included data preprocessing and data segmentation, followed by a two-stage approach. In Stage 1, we estimated the probability of each inertial data frame being intake or non-intake, benchmarking different deep learning models and architectures. Based on the probabilities estimated in Stage 1, we detected the intake gestures in Stage 2 and calculated the F1 score for each model. Results indicate that top model performance was achieved by a CNN-LSTM with earliest sensor data fusion through a dedicated CNN layer and a target matching technique (F 1 = .778). As for data preprocessing, results show that applying a consecutive combination of mirroring, removing gravity effect, and standardization was beneficial for model performance, while smoothing had adverse effects. We further investigate the effectiveness of using different combinations of sensor modalities (i.e., accelerometer and/or gyroscope) and sensor positions (i.e., dominant intake hand and/or non-dominant intake hand).

Highlights

Advances in mobile sensor technologies have enabled novel forms of dietary assessment
While dietary assessment was traditionally carried out exclusively using active methods for capturing food intake based on human effort to collect data (e.g., 24-hr recalls, food records), passive capture methods aim to reduce burden on individuals associated with collecting dietary data by using a range of different sensor technologies
(2) Proposed Model and Benchmarking: We propose a new model that achieved better performance (F1 = .778) compared to current state-of-the-art deep learning models for detecting intake gestures based on inertial data, using our novel large-scale dataset

Summary

Introduction

Advances in mobile sensor technologies have enabled novel forms of dietary assessment. While dietary assessment was traditionally carried out exclusively using active methods for capturing food intake based on human effort to collect data (e.g., 24-hr recalls, food records), passive capture methods aim to reduce burden on individuals associated with collecting dietary data by using a range of different sensor technologies (e.g., inertial measurement units, microphones, The associate editor coordinating the review of this manuscript and approving it for publication was Gang Mei. and video cameras). Sensor technologies have the potential of complementing active capture methods for quantifying food intake [1] (e.g., by verifying intake activities, prompting human capture). The wrist-worn Inertial Measurement Unit (IMU) has emerged as a promising technology for sensor-based passive capture of food intake [2]–[4]. Triaxial accelerometers in IMUs measure changes in speed and

Methods

Results

Conclusion