Human activity recognition based on smartphone and wearable sensors using multiscale DCNN ensemble

Abstract

Sensor-based Human Activity Recognition (sensor-based HAR) has been used in many real-world applications providing valuable knowledge to many areas, such as human-object interaction, medical, military and security. Recently, wearable devices have progressively gained momentum due to their relevant data provided by their sensors, which could be employed in sensor-based HAR. In addition, the large number of sensors present in these devices provides complementary data since each sensor provides distinct information. However, there are two main issues: data heterogeneity between multiple sensors and the temporal nature of the sensor data. To cope with the former issue, we process each sensor separately, learning their features and performing the classification before fusing with the other sensors. To exploit the latter issue, we use an approach to extract patterns in multiple temporal scales of the data, using an ensemble of Deep Convolution Neural Networks (DCNN). This is convenient since the data are already a temporal sequence and the multiple scales extracted provide meaningful information regarding the activities performed by the users. Consequently, our approach is able to extract both simple movement patterns, such as a wrist twist when picking up a spoon and complex movements, such as the human gait. This multimodal and multi-temporal approach outperforms previous state-of-the-art works in seven important datasets using two different protocols. Finally, we demonstrate that our proposed set of kernels improves sensor-based HAR in another multi-kernel approach, the widely employed inception network.