Emotional analysis of joint sports quality expansion tasks based on multi-modal feature fusion

Abstract

A multi-modal feature based motion emotion analysis model based on a fusion deep learning model is proposed for the problem of analyzing the motion emotions of participants in the joint exercise quality expansion task. This model involves three major modalities: EEG signals, peripheral physiological signals, and facial expression signals, and processes and fuses the information of these three main modalities to achieve the effect of processing multi-dimensional motor emotional information. At the same time, this study introduces the design concept of residual networks, using self attention modules and multi head mutual attention modules to process different modal features. The results showed that the combination of peripheral physiological modality and facial expression modality had the highest accuracy among the three modality combinations, with an accuracy rate of 88.8 %. The feature fusion method based on the cascaded residual attention mechanism module has better accuracy and F1 Score performance than other methods. In addition, different emotional states can be effectively identified and distinguished in these three modalities, indicating that the model has a wide range of possibilities in practical applications.