Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition

Abstract

Human behavior analysis from big multimedia data has become a trending research area with applications to various domains such as surveillance, medical, sports, and entertainment. Facial expression analysis is one of the most prominent clues to determine the behavior of an individual, however, it is very challenging due to variations in face poses, illuminations, and different facial tones. In this paper, we analyze human behavior using facial expressions by considering some famous TV-series videos. Firstly, we detect faces using Viola-jones algorithm followed by tracking through Kanade-Lucas-Tomasi (KLT) algorithm. Secondly, we use histogram of oriented gradients (HOG) features with support vector machine (SVM) classifier for facial recognition. Next, we recognize facial expressions using the proposed light-weight convolutional neural network (CNN). We utilize data augmentation techniques to overcome the issue of appearance of faces from different views and lightening conditions in video data. Finally, we predict human behaviors using an occurrence matrix acquired from facial recognition and expressions. The subjective and objective experimental evaluations prove better performance for both facial expression recognition and human behavior understanding.