Correlation mining of multimodal features based on higher-order partial least squares for emotion recognition in conversations

Abstract

In fields requiring an understanding of emotions, such as digital human interaction and public opinion analysis, achieving a dependable and interpretable model for mining correlations among multimodal features remains a primary objective. However, current deep learning methods often lack transparency and suffer from low interpretability. To address these challenges, we propose a novel Correlation Mining method based on Higher-Order Partial Least Squares (HOPLS) for multimodal Emotion Recognition in conversations (CMHER) in this paper. CMHER innovatively combines HOPLS with transformers and Gated Recurrent Units (GRUs) to compute correlation matrices within unimodal data streams and between cross-modal sources. HOPLS projects source data into a latent space to predict target data via correlation matrix computations, eliminating the need for Graphical Processing Unit (GPU) acceleration and making it suitable for experimental and edge systems. The integration of HOPLS with deep neural networks involves preprocessing multimodal features into suitable dimensions and latent representations, followed by HOPLS computing correlation matrices for cross-modal latent vectors and final labels through optimal joint subspace approximation, which aims at the improvements of both interpretability and reliability. Additionally, a generalization error fitting module further refines the predicted correlation matrices to improve predictive capability and overall model performance. Experiments on two public datasets validate the superiority of our proposed method.