Multimodal sensor fusion models for real-time exercise repetition counting with IMU sensors and respiration data

Abstract

This study introduces an automated exercise repetition counting model for a variety of exercises. Although external devices such as fitness trackers and smart watches are capable of tracking exercise progress, they are still incapable of recording all pertinent information essential for comprehensive exercise tracking, such as the number of repetitions.To address this limitation, we propose a deep learning model that utilizes multimodal sensor and respiration audio data obtained from wearable devices, specifically smart earbuds that record IMU sensors and collect surrounding sounds. A diverse dataset of sensor data and respiratory audio data are collected by leveraging these smart earbuds during 30 different types of exercises, which enables the study to develop a comprehensive and accurate multimodal deep learning model that can be applied to a wide range of physical activities. Advanced techniques such as data augmentation and time-series interpolation are also employed to address the challenges of working with large and complex datasets.The proposed deep learning model, which incorporates multimodal sensor data and advanced data processing techniques, can be utilized to promote more effective and personalized exercise plans, ultimately leading to improved health outcomes for individuals who seek to maintain an active and healthy lifestyle.