CLASSIFICATION OF EMOTIONAL RESPONSES IN VIDEO AND AUDIO RECORDINGS USING A CONVOLUTIONAL NEURAL NETWORK

Abstract

Depression and depressive symptoms are associated in Alzheimer disease (AD) (Green et. al, Arch. Neurol., 2003). Depressive symptoms have been reported as occurring prior to the development of AD dementia and could be a potential indicator of impending dementia. Hence, identifying emotions that reflect underlying depressive symptoms may provide an early risk score estimate as well as serve as a quality control measure to help physicians make better decisions. Deep Learning is an application of Artificial Neural Networks with more than one hidden layer. It has been shown that with increasing data, the performance of deep learning increases significantly. Convolutional Neural Network (CNN) in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, has proven to be highly successful in image classification. Audio features have been minimally used and integrated for video classification problems. Here we combine audio and visual features in a unique fashion to train the machine learning model and make the predictions. The dataset (Jiang et al, AAAI, 2014) labelled with 8 emotions namely ‘Anger’, ‘Anticipation’, ‘Disgust’, ‘Fear’, ‘Joy’, ‘Sadness’, ‘Surprise’ and ‘Trust’ was obtained. The model was trained using a CNN merging the video images and the corresponding audio spectrograms. The CNN was trained on pre-trained inceptionV3 model by adding a fully connected layer with 1024 neurons and a “softmax” output layer. The prototype model from the methodology described above obtained an accuracy of approximately 85% in correct classification of the emotions. We are currently in the process of verifying the result, and obtaining the Audio/Visual Emotion Challenge (AVEC) datasets – a challenge aiming to provide common benchmark test set, and to bring together research communities, to establish fusion of possible and beneficial approaches for depression and emotion analysis, to validate our model.