A deep learning framework for heart rate estimation from facial videos

Abstract

Accurate heart rate is vital to acquiring critical physical data of human subjects. For this reason, facial video-based heart rate estimation has recently received tremendous attention owing to its simplicity and convenience. However, its accuracy, reliability and computational complexity have yet to reach the desired standards. In this work, we have endeavored to develop a novel deep learning framework for real-time estimation of heart rates by using an RGB camera. Our approach consists of the following four steps. We begin Step 1 by detecting the face and facial landmarks in the video to identify the required facial Region of Interests (ROIs). In Step 2, we extract the sequence of the mean of the green-channeled video from the facial ROIs, and explore a three-stage sequential filtering, including illumination rectification, trend removal and signal amplification. In Step 3, the Short-Time Fourier Transform (STFT) is employed to convert the 1D filtered signal into the corresponding 2D Time-Frequency Representation (TFR) for characterizing the frequencies over short time intervals. The 2D TFR allows the formulation of the heart rate estimation as a video-based supervised learning problem, which can be solved by exploring a deep Convolutional Neural Network (CNN), as is carried out in Step 4. Our approach is one of the pioneering work that proposes a deep learning framework with TFRs as input for solving the heart rate estimation from facial videos. In addition, we have developed a heart rate database, named the Pulse From Face (PFF), and used it along with the existing PURE (PUlse RatE) database [1] to train the CNN. The PFF database is released for research purpose with this paper. We have evaluated the proposed framework on the MAHNOB-HCI database [2] and the VIPL-HR database [3] and compared its performance with that of other contemporary approaches to demonstrate its efficacy.