A new pipeline for the recognition of universal expressions of multiple faces in a video sequence

Abstract

Facial Expression Recognition (FER) is a crucial issue in human-machine interaction. It allows machines to act according to facial expression changes. However, acting in real time requires recognizing the expressions at video speed. Usually, the video speed differs from one device to another. However, one of the standard settings for shooting videos is 24 fps. This speed is considered as the low-end of what our brain can perceive as fluid video. From this perspective, to achieve a real-time FER the image analysis must be completed, strictly, in less than 0.042 second no matter how the background complexity is or how many faces exists in the scene. In this paper, a new pipeline has been proposed in order to recognize the fundamental facial expressions for more than one person in real world sequence videos. First, the pipeline takes as input a video and performs a face detection and tracking. Regions of Interest (ROI) are extracted from the detected face in order to extract the shape information when applying the Histogram of Oriented Gradient (HOG) descriptor. The number of features yield by HOG de-scriptor is reduced by means of a Linear Discriminant Analysis (LDA). Then, a deep data analysis was carried out, exploiting the pipeline, for the objective of setting up the LDA classifier. The analysis aimed at proving the suitability of the decision rule selected to separate the facial expression clusters in the LDA training phase. To conduct our analysis, we used ChonKanade(CK+) database and F-measure as an evaluation metric to calculate the average recognition rates. An automatic evaluation over time is proposed, where labelled videos is utilized to investigate the suitability of the pipeline in real world condition. The pipeline results showed that the use of HOG descriptor and the LDA gives a high recognition rate of 94.66%. It should be noted that the proposed pipeline achieves an average processing time of 0.018 second, without requiring any device that speeds up the processing.