Deep neural networks for multimodal data fusion and affect recognition

Abstract

This paper proposes novel deep neural network models to handle multimodal data. The proposed models seamlessly facilitate fusion of multimodal inputs and bring about dimensional reduction of the input feature space. The architecture employs multimodal stacked autoencoder in conjunction with multi-layer perceptron-based regression model. Two variants of the architecture are proposed. Experiments have been performed on the multimodal benchmark dataset (RECOLA) to illustrate the importance of multimodality for affect recognition. The proposed architectures are trained using effective training strategies, specifically designed to reduce the number of tuneable parameters for multimodal applications. The results obtained are encouraging and the proposed approach is computationally less expensive than the existing approaches. The performance is better or at par with the other techniques.