Abstract
Emotion information from speech can effectively help robots understand speaker's intentions in natural human-robot interaction. The human auditory system can easily track temporal dynamics of emotion by perceiving the intensity and fundamental frequency of speech, and focus on the salient emotion regions. Therefore, speech emotion recognition combined with the auditory mechanism and attention mechanism may be an effective way. Some previous studies used auditory-based static features to identify emotion while ignoring the emotion dynamics. Some other studies used attention models to capture the salient regions of emotion while ignoring cognitive continuity. To fully utilize the auditory and attention mechanism, we first investigate temporal modulation cues from auditory front-ends and then propose a joint deep learning model that combines 3D convolutions and attention-based sliding recurrent neural networks (ASRNNs) for emotion recognition. Our experiments on the IEMOCAP and MSP-IMPROV datasets indicate that the proposed method can be effectively used to recognize the emotions of speech from temporal modulation cues. The subjective evaluation shows that the attention patterns of the attention model are basically consistent with human behaviors in recognizing the emotions.
Highlights
Speech is the most natural way for communication between humans and robots
The IEMOCAP dataset consists of five sessions, where each session contains scripted and improvised utterances from two speakers
Taking into account, that the human auditory system has a very strong ability to perceive the intensity and fundamental frequency of speech, it can track temporal dynamics of emotion from the perceived information and focus on the salient emotion regions, we propose a speech emotion recognition (SER) system by combining auditory mechanism and attention mechanism of human auditory system
Summary
Speech is the most natural way for communication between humans and robots. The key point of effective communication is to make robots or virtual agents understand speakers’ true intentions. Only using the linguistic information is by no means sufficient enough for understanding of intentions. The vocal emotion information as a kind of nonlinguistic information can significantly help robots or virtual agents to understand speakers’ true intentions. Speech emotion recognition (SER) is the research hotspot in natural human-robot interaction (HRI). Effective SER is still a very challenging problem, partly due to
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have