Machine Learning Activity Classification Models For Preschool-aged Children: The Need For Free-living Training Data.

Abstract

Machine learning (ML) classification models trained on laboratory activity trials exhibit poor performance when evaluated under free-living conditions. Training models on free living data, including temporal features such as lead and lag windows, and using shorter sliding windows may improve recognition accuracy under free-living conditions. PURPOSE: To evaluate the accuracy of free-living hip and wrist Random Forest activity classifiers for pre-schoolers trained on features extracted from windows of 1s, 5s, 10s, and 15s. Performance was benchmarked against classifiers trained on laboratory-based data using a 15s window. METHODS: 31 preschool-aged children (4.0 ± 0.9 y) were video recorded during a 20-minute unstructured active play session. Participants wore an accelerometer on their right hip and non-dominant wrist. A bespoke two-stage direct observation system was used to code ground truth activity class and specific activity types occurring within each class. Data from 21 of the children were randomly selected to train the classifiers. Models were trained with and without temporal features and cross-validated in a hold sample of the remaining 10 children with overall and class-level accuracy. RESULTS: Accuracy improved as window size increased from 1 sec (73.5%-77.7%) to 10 sec (82.4%-86.0%); with only minimal improvements observed for 15s windows. Inclusion of lag and lead features increased accuracy by 1.6% to 6.6%, with the largest improvements observed for shorter duration windows (≤10s). Comparatively, the accuracy of the laboratory trained model was 56.9% and 67.5% for wrist and hip, respectively. For a 10s window, training models on free-living data and including temporal features increased recognition of sedentary from 70.6% - 74.4% to 83.3% - 90.4%; light activities and games from 57.5% - 76.9% to 88.6% - 88.8%; walking from 7.5% - 17.5% to 64.1% - 75.0%; and running from 50.0% - 77.8% to 71.4% - 85.7%. There was no improvement in recognition of mod-vig activities and games (56.3% - 62.5%). CONCLUSIONS: Unlike models trained on laboratory activity trials, ML activity classification models for pre-schoolers trained on free-living accelerometer data perform well when evaluated under true free-living conditions. Funding: Australian Research Council Discovery Project Grant: DP150100116