Multimodal Emotion Recognition Using Classifier Reliability-Based Aggregation

Abstract

This paper addresses emotion recognition by first individually processing and then aggregating different modes of human communication through a classification and aggregation framework. Specifically, the proposed framework processes speech acoustics, facial expressions, and body language using unimodal emotion classifiers. The speech emotion is classified using a deep neural network (DNN) while facial and body language emotion classifiers are implemented using classifiers implemented through supervised fuzzy adaptive resonance theory. The speech emotion classifier uses acoustic features, the facial emotion classifier uses features based on facial animation parameters (FAP), and body language emotion classifier uses head and hands features. The unimodal evaluations are then aggregated this paper also proposes classifier reliability-based aggregation preferences for the unimodal evaluations. The reliability-based preferences are extracted from the accuracies of the unimodal classifiers for each emotion. The results show that the proposed framework outperforms the existing techniques. Furthermore, because of late fusion, the functionality of the proposed approach is robust to unavailability of all but one of the modes of communication.