Evaluating and Enhancing the Generalization Performance of Machine Learning Models for Physical Activity Intensity Prediction From Raw Acceleration Data.

Vahid Farrahi,Maisa Niemela,Raija Korpelainen,Maarit Kangas,Petra Tjurin,Timo Jamsa

doi:10.1109/jbhi.2019.2917565

Abstract

To evaluate and enhance the generalization performance of machine learning physical activity intensity prediction models developed with raw acceleration data on populations monitored by different activity monitors. Five datasets from four studies, each containing only hip- or wrist-based raw acceleration data (two hip- and three wrist-based) were extracted. The five datasets were then used to develop and validate artificial neural networks (ANN) in three setups to classify activity intensity categories (sedentary behavior, light, and moderate-to-vigorous). To examine generalizability, the ANN models were developed using within dataset (leave-one-subject-out) cross validation, and then cross tested to other datasets with different accelerometers. To enhance the models' generalizability, a combination of four of the five datasets was used for training and the fifth dataset for validation. Finally, all the five datasets were merged to develop a single model that is generalizable across the datasets (50% of the subjects from each dataset for training, the remaining for validation). The datasets showed high performance in within dataset cross validation (accuracy 71.9-95.4%, Kappa K = 0.63-0.94). The performance of the within dataset validated models decreased when applied to datasets with different accelerometers (41.2-59.9%, K = 0.21-0.48). The trained models on merged datasets consisting hip and wrist data predicted the left-out dataset with acceptable performance (65.9-83.7%, K = 0.61-0.79). The model trained with all five datasets performed with acceptable performance across the datasets (80.4-90.7%, K = 0.68-0.89). Integrating heterogeneous datasets in training sets seems a viable approach for enhancing the generalization performance of the models. Instead, within dataset validation is not sufficient to understand the models' performance on other populations with different accelerometers.

Highlights

A CCELEROMETERS are small, reliable, and feasible tools for objective measurement of physical activity (PA) [1]
The results demonstrated that merging data from hip and wrist accelerometers collected from various population groups monitored with different devices is a viable approach to enhance the generalization performance of artificial neural networks (ANN) models in PA intensity classification across different population groups monitored by a single hip- or wrist-worn accelerometer
According to a previous research showing the effects of data acquisition protocols on the performance of machine learning (ML)-based models [20], [37], the mixed results for the classification accuracy of SB and MVPA across the datasets with similar placement is perhaps due to differences in data acquisition protocols and performed activities in the datasets

Summary

Introduction

A CCELEROMETERS are small, reliable, and feasible tools for objective measurement of physical activity (PA) [1]. Different PA types, energy expenditure, and intensities can be assessed from acceleration signals [1]. Classification of acceleration data across the whole intensity spectrum is one of the most common measures for a variety of studies including clinical, surveillance, and intervention studies [2]. Intensity classification of activities has been performed using cut-point-based methods that have been established for both activity counts and raw acceleration data [1], [2]. The accuracy of the cut-points has been reported to be limited [1], [3]. Raw accelerometry and machine learning (ML) modeling approaches have been used for both standardization and harmonization of accelerometry results and

Objectives

Results

Discussion

Conclusion