Feature Extraction and Representation Techniques for Facial Expression Analysis

Abstract

With the rapid development of applications related to human-computer interaction, facial expression recognition plays an important role in affective computing technologies and can benefit many applications in computer technology, security, behavioural research, and clinical investigations on patients with neuropsychiatric disorders. This research aims towards developing algorithms and frameworks for facial expression recognition with a low computational complexity, which are suitable for real-time applications. Instead of developing a real-time system for a specific application, developing the components of facial expression recognition systems is the main focus of this research. One of the significant components of a facial expression recognition system is facial feature extraction. Two feature extraction algorithms, based on appearance and geometry, were developed for image sequences. To assess the performance of these algorithms, simulations were carried out on various datasets containing expressions of different intensities (e.g. apparent and subtle expressions) and complexity. The proposed appearance-based algorithm, known as Spatio-Temporal Texture Map (STTM), demonstrated its capability to extract subtle motions of facial expressions and attained superior performances with a low computational cost. Similarly, the proposed geometry-based feature extraction algorithm, based on Active Appearance Model (AAM), demonstrated its excellent performance in annotating landmark points on videos. It was found that the number of iterations required in AAM fitting could be reduced by updating the parameters frame-by-frame. Moreover, to take into account the temporal information of a video, neighbouring frames were considered in AAM fitting which improved the annotation performance. The geometric feature is further evaluated for facial expression recognition and showed an excellent performance in this task. Keeping in mind the significance of the dynamics of facial expressions, intensity estimation of facial expressions is also addressed in this work. Another problem addressed is expressions accompanied by head movements which makes decoding the depicted expressions difficult. Even though research in facial expression recognition has been active since the last two decades, these topics recently gained attention from researchers. In order to address the former problem, a framework which jointly recognizes facial expressions and estimates facial expression intensities from image sequences was developed. The framework consists of k Nearest Neighbours (kNN), a weighting scheme, and a change-point detector. With low computational complexity, the proposed algorithm showed its superior performance, especially in estimating facial expression intensities. To address the latter problem, where the facial expression is captured at several different angles, a representation for multi-view facial expression recognition was developed. The proposed algorithm showed its excellent performance based on the evaluations against state-of-the-art algorithms.