Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Hyeokhyen Kwon,Thomas Plötz,Gregory D Abowd

doi:10.3390/s21248337

Hyeokhyen Kwon, Thomas Plötz + Show 1 more

Open Access

https://doi.org/10.3390/s21248337

Copy DOI

Journal: Sensors	Publication Date: Dec 13, 2021
Citations: 14	License type: CC BY 4.0

Affiliation: Georgia Institute of Technology, Northeastern University

Abstract

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

Highlights

Human activity recognition based on wearable sensing platforms (HAR) is a core component of mobile, ubiquitous, and wearable computing
We aim to demonstrate that human activity recognition (HAR) models with significantly increased model complexity compared to the popular DeepConvLSTM model can be effectively trained by making use of IMUTube for generating large-scale virtual inertial measurement units (IMUs)
We incrementally applied the individual modules as they were introduced in Section 3 to explore the overall effectiveness of the new model architecture and the impact each part has on the recognition accuracy

Summary

Introduction

Human activity recognition based on wearable sensing platforms (HAR) is a core component of mobile, ubiquitous, and wearable computing. Miniaturized inertial measurement units (IMUs), integrated into either body-worn devices such as smart watches, fitness bands, or head-worn units, or mobile devices such as smart phones are used to capture a person’s movements These movement signals are automatically analyzed to recognize and assess activities that are of relevance for many practical applications, including, for example, gesture-based interaction [1], health assessments [2,3], or behavioral authentication [4]. Unlike in other application domains of machine learning, here it is challenging to collect large amounts of correctly annotated data samples. The labeled datasets, that are needed for supervised model training, are too small—a challenge we refer to as the “small data(set)” problem.

Objectives

Results

Discussion

Conclusion