Three-Stream Convolutional Neural Network for Human Fall Detection

Abstract

Lower child mortality rates, advances in medicine, and cultural changes have increased life expectancy to above 60-years old in developed countries. Some countries expect that, by 2030, 20% of their population will be over 65 years old. The quality of life at this advanced age is highly dictated by the individual’s health, which will determine whether the elderly can engage in important activities to their well-being, independence, and personal satisfaction. Old age is accompanied by health problems caused by biological limitations and muscle weakness. This weakening facilitates the occurrence of falls, which are responsible for the deaths of approximately 646,000 people worldwide and, even when a minor fall occurs, it can still cause fractures, break bones, or damage soft tissues, which will not heal completely. Injuries and damages of this nature, in turn, will consume the self-confidence of the individual, diminishing their independence. In this work, we propose a method capable of detecting human falls in video sequences using multi-channel convolutional neural networks (CNN). Our method makes use of a 3D CNN fed with features previously extracted from each frame to generate a vector for each channels. Then, the vectors are concatenated, and a support vector machine (SVM) is applied to classify the vectors and indicate whether or not there was a fall. We experiment with four types of features, namely: (i) optical flow, (ii) visual rhythm, (iii) pose estimation, and (iv) saliency map. The benchmarks used (UR Fall Detection Dataset (URFD) [] and (ii) Fall Detection Dataset (FDD) []) are publicly available and our results are compared to those in the literature. The metrics selected for evaluation are balanced accuracy, accuracy, sensitivity, and specificity. Our results are competitive with those obtained by the state of the art on both URFD and FDD datasets. To the authors’ knowledge, we are the first to perform cross-tests between the datasets in question and to report results for the balanced accuracy metric. The proposed method is able to detect falls in the selected benchmarks. Fall detection, as well as activity classification in videos, is strongly related to the network’s ability to interpret temporal information and, as expected, optical flow is the most relevant feature for detecting falls.