An Accurate and Explainable Multimodal Deep Fusion Network for Affect Recognition

Abstract

Affective analysis is an emerging research area which helps human to better understand their mental state through human-machine interaction. During the interaction process, bio-signal analysis is the essential work of detecting human affective changes since bio-signals are considered as a representation of physiological responses which are related to human affective states. Machine learning methods to analyse bio-signals are currently promoted as the better way to detect physiological changes, but most empirical works have utilised limited types of bio-signals in affect recognition, which cannot provide precise results. Moreover, these empirical works have mainly deployed traditional machine learning methods rather than deep learning models, which may have the opportunity to improve the classification accuracy.This research provides a performance comparison between traditional machine learning models and deep learning models on multimodal bio-signals for human mental state classification tasks. The extensive experimental results suggest that deep learning algorithms outperform traditional machine learning algorithms in accuracy and weighted F1 score for classification tasks in affective analysis.Furthermore, to improve the explainability of the deep learning model, this research conducted a thorough analysis to understand the contribution of each bio-signal, and how they differ for various affective states. The research work improves the state of the art for emotion recognition from bio-signals and the current understanding of the relationship between affect and bio-signals.