Learning long-term temporal contexts using skip RNN for continuous emotion recognition

Abstract

BackgroundContinuous emotion recognition as a function of time assigns emotional values to every frame in a sequence. Incorporating long-term temporal context information is essential for continuous emotion recognition tasks. MethodsFor this purpose, we employ a window of feature frames in place of a single frame as inputs to strengthen the temporal modeling at the feature level. The ideas of frame skipping and temporal pooling are utilized to alleviate the resulting redundancy. At the model level, we leverage the skip recurrent neural network to model the long-term temporal variability by skipping trivial information for continuous emotion recognition. ResultsThe experimental results using the AVEC 2017 database demonstrate that our proposed methods are beneficial to a performance improvement. Further, the skip long short-term memory (LSTM) model can focus on the critical emotional state when training the models, thereby achieving a better performance than the LSTM model and other methods.