Emotion recognition based on 3D fuzzy visual and EEG features in movie clips

Abstract

In this paper, we propose an emotion recognition system for understanding the emotional state of humans while watching a movie clip. Using various movies clips with scene changes over time, a 3D fuzzy GIST was used for dynamic emotional feature extraction from low-level visual features, and a 3D fuzzy tensor was used for semantic-level brain features related to the emotional state of humans. In the case of dynamic visual features, the 3D fuzzy GIST consists of L⁎C⁎H⁎ color (L: Lightness; C: Chroma; and H: Hue) and orientation information of a movie clip in a predefined time interval. For dynamic brain features, we processed the electroencephalographic (EEG) signals, as stimulated by the movie clips to induce an emotional state, through both an independent component analysis (ICA) to eliminate artifacts, and Short Time Fourier Transform (STFT) to extract the reliable features. To obtain a hemisphere power asymmetry, the 3D tensor data for the brain signals were constructed according to the time-dependent energy at the alpha band (8–13Hz) and gamma band (30–60Hz). Finally, the 3D fuzzy GIST and 3D fuzzy tensor were obtained through fuzzy C-means clustering using visual and EEG signals, respectively. The obtained 3D fuzzy GIST and 3D fuzzy tensor features were used as inputs to an adaptive neuro-fuzzy inference (ANFIS) classifier, which was provided using the mean opinion scores (MOSs) as the teaching signals. Experimental results show that, using an ANFIS classifier, the proposed 3D fuzzy visual and EEG features are effective in building an emotion recognition system.