A novel pedal musculoskeletal response based on differential spatio-temporal LSTM for human activity recognition

Abstract

Human activity recognition (HAR) with wearable devices has shown superior usability in everyday life tracking and healthcare monitoring in recent years. To solve the existing challenges of weak sensor durability and difficulty capturing dynamic features of traditional wearable devices based on the long short-term memory neural network (LSTM), a new HAR system based on a pedal wearable device is proposed by utilizing a novel differential spatio-temporal LSTM (DST-LSTM) method. The pedal wearable device is embedded in the tongue area of the general footwear to obtain pedal musculoskeletal response (PMR) data of dorsum pedis. Based on the PMR data, the DST-LSTM is developed to classify five typical activity statuses as standing, sitting, floor walking, down the stairs and up the stairs. First, the dynamic differential information of the PMR data is taken into consideration to build a new LSTM unit. Second, the spatial relationship features of the PMR data are obtained by multi-head graph attention networks (GAT). Third, a novel spatial gate is added to the new LSTM unit to filter irrelevant spatial features and get valid information. The effectiveness and superiority of both the pedal wearable device and the DST-LSTM method are validated by comparison with the state-of-the-art methods and wearable devices.