Multimodal emotion detection from multiple data streams for improved decision making

Abstract

Recent neurological studies shows that emotions are tightly connected to the thinking and cognitive actions, being part of the decision-making process. Considering this, having a way to help decision making processes based on current emotion of the user or to consider the potential emotional impact if a decision is made, would be beneficial. This paper introduces a novel method for fusing multiple emotional signals, using a weighted average, where each weight value adapts to real time conditions, based on signal type, presence, and quality. In the context of a training station for manual operation, we implemented and tested separately several emotion detection methods, each based on a different signal acquired from audio, video, and galvanic skin response data streams. The final goal is to include the proposed method together with state of the art emotion detection machine learning algorithms as part of the digital twin training station for manual operation.