A Customized Convolutional Neural Network Model Integrated With Acceleration-Based Smart Insole Toward Personalized Foot Gesture Recognition

Abstract

Foot gestures play an important role in human machine interface and also indicate the lower-limb motor functionalities. Lately, machine learning models have been integrated with different kinds of wearable devices for foot gesture recognition. However, most existing wearable devices lack of robustness and facileness and existing machine learning models suffer from inconsistent recognition accuracy due to limited and unbalanced training datasets and, therefore, do not work well for new users. To address this, a customized convolutional neural network model architecture is designed, and strategies for achieving personalized models are presented, considering potential fairness sensitive attributes of a diverse training group, including age, gender, body mass index (defined by height and weight), shoe size, and health conditions. The proposed model to be embodied into a flexible smart insole with only two accelerometers aims to recognize four types of foot gestures-toe tapping, heel tapping, foot kicking, and foot stepping-in a sitting posture at high accuracies. After the hyperparameters are well-tuned, average recognition accuracy reaches 92% in training sets, 88.05% in validation sets, 85.27% in test sets, and 83.09% in fivefold cross-validation. The evaluation results from both in-group and leave-one-subject-out show that our unified model is easily transformable for individual uses by further fine-tuning the key hyperparameters.