Bimodal feature-based fusion for real-time emotion recognition in a mobile context

Abstract

This research explores the viability of a bimodal fusion of linguistic and acoustic cues in speech to help in realtime emotion recognition in a mobile application that steers the interaction dialogue in tune with user's emotions. For capturing affect at the language level, we have utilized both, machine learning and valence assessment of the words carrying emotional connotations. The indicative values of acoustic cues in speech are of special concern in this research and an optimized feature set is proposed. We highlight the results of both independent evaluations of the underlying linguistic and acoustic processing components. We present a study and ensuing discussion on the performance metrics of a Logistic Model Tree that has outperformed the other classifiers considered for the fusion process. The results reinforce the notion that capturing the sound interplay between the diverse set of features is crucial for confronting the subtleties of affect in speech that so often elude the text- or acoustic-only approaches to emotion recognition.