Visualization of time-varying joint development of pitch and dynamics for speech emotion recognition

Abstract

In this paper, we propose a new approach for visualizing the time-varying acoustic features for speech emotion recognition. Although the emotional state does not carry any linguistic information, it is a crucial factor that offers sentiment feedback to the listener. We propose to extract the two most prevalent acoustic features: pitch and dynamics, to identify the speech emotion of the speaker. We represent the time-varying pitch and dynamics as a trajectory in a two-dimensional feature space. Multiple trajectories are then segmented and clustered into signature patterns. This technique was successful in identifying and retargeting expressive musical performance styles. In evaluation, we use the German emotion language database. The database was created with ten professional actors (five males and five females) of ten emotionally unbiased sentences performed in six target emotions (Angry, Happy, Fear, Boredom, Sad, and Disgust). Results showed that the speech samples from the same actor of the same sentence but different emotions have dramatically different trajectory patterns. On the other hand, obvious common patterns were found among low valence emotions like Boredom and Sadness. The current study also opens future research opportunities for applying advanced pattern recognition techniques (e.g., Support Vector Machine and Neural Network) for better emotion identification.