Sitting is the new smoking: online complex human activity recognition with smartphones and wearables

Abstract

Human activity recognition plays an important role in fitness tracking, health monitoring, context-aware feedback and self-management of smartphones and wearable devices. These devices are equipped with different sensors which can be used to recognize various human activities. A significant amount of work has been done in human activity recognition using such devices by different researchers. However, most of the work focuses on simple physical activities such as walking, jogging, biking, writing, typing, sitting and standing. The recognition of such complex activities still needs to be further explored. Complex activities may involve hand gestures which are not periodic; for example, eating, drinking coffee, smoking, and giving a talk. Also, most of the existing work has been performed offline (not on the device). In the online method, this process is performed in real-time on the device. In this context, we investigate the recognition of both simple and complex activities using different sensors from smartphones and wearables. To this end, we address the following question: How to recognize various human activities using different sensors from wearable devices and smartphones both offline and online (on the device)? In this context, we collected multiple datasets. Using these datasets, we investigate the recognition of both simple and complex human activities using various machine learning algorithms. Based on this analysis, we provide recommendations on how and when to use certain sensors, classifiers and body positions for the recognition of a specific activity. We also propose to use a hierarchical lazy classification approach for the recognition of complex activities involving hand gestures such as smoking and other similar activities. We developed an online activity recognition framework for smartphones and smartwatches. Based on this framework, we implement a prototype application for these devices which can recognize various human activities in real-time. As an example use case, we use the smartphone for recognizing seven physical activities, whereas the smartwatch is used for smoking recognition. We also investigate the resource consumption (CPU, memory, and power) of our online activity recognition system on a mobile phone and a smartwatch with respect to different aspects.