SDTF-Net: Static and dynamic time–frequency network for Speech Emotion Recognition

Abstract

Speech Emotion Recognition (SER) is an important but challenging research topic in Human–Computer interaction (HCI). As a crucial component influencing the model’s performance, in SER, feature extraction needs to comprehensively consider the static local states, the dynamic change tendency, and the time–frequency characteristics of speech signals to generate meaningful emotional representations and further improve the SER performance. To address this aim, we propose the Static and Dynamic Time–frequency Network called SDTF-Net. To reduce the interference between different features, in SDTF-Net, we introduce two parallel modules, i.e., Time–Frequency Convolutional Neural Network (TF-CNN) and Time–Frequency Transformer (TF-Trans), to generate the static and dynamic emotional features based on the time–frequency information, respectively. Specifically, in the TF-Trans module, we design a three-step strategy to learn the relatively complicated dynamic feature, comprising two parallel Bi-direction Multi-scale Dynamic Activation (BMDA) blocks, two parallel transformer (Trans) blocks, and a time–frequency fusion (TF-Cross) block. Subsequently, we propose an SD-Cross module to effectively fuse the learned static and dynamic features, generating the final meaningful emotional representations. Given the problem of data sparsity, moreover, we employ the data augmentation method called SpecAugment to improve the learning ability of SDTF-Net. From the experimental results on IEMOCAP and MSP-IMPROV datasets, we observe that SDTF-Net outperforms the state-of-the-art methods. Furthermore, extensive ablation experiments further indicate the rationality of the SDTF-Net structure and the necessity of considering dynamic and static features based on time–frequency information comprehensively in SER.