K-Means Clustering-Based Kernel Canonical Correlation Analysis for Multimodal Emotion Recognition in Human–Robot Interaction

Abstract

In this article, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -meansclustering-based Kernel canonical correlation analysis algorithm is proposed for multimodal emotion recognition in human–robot interaction (HRI). The multimodal features (gray pixels; time and frequency domain) extracted from facial expression and speech are fused based on Kernel canonical correlation analysis. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -means clustering is used to select features from multiple modalities and reduce dimensionality. The proposed approach can improve the heterogenicity among different modalities and make multiple modalities complementary to promote multimodal emotion recognition. Experiments on two datasets, namely SAVEE and eNTERFACE‘05, are conducted to evaluate the accuracy of the proposed method. The results show that the proposed method produces good recognition rates that are higher than the ones produced by the methods without <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -means clustering; more specifically, they are 2.77% higher in SAVEE and 4.7% higher in eNTERFACE‘05.