ReSenseNet: Ensemble Early Fusion Deep Learning Architecture for Multimodal Sentiment Analysis

Abstract

AbstractMultimodal sentiment analysis is an actively emerging field of research in deep learning that deals with understanding human sentiments based on more than one sensory input. In this paper, we propose reSenseNet, an ensemble of early fusion architecture of deep convolutional neural network (CNN) and Long Short term Memory (LSTM) for multimodal sentiment analysis of audio, visual, and text data. ReSenseNet consists of feature extraction, feature fusion, and fully connected layers stacked together as a three-layer architecture. Instances of the generalized reSenseNet architecture have been experimented on several variants of modalities combined together to form different variations in the test data. Such a combination has produced results in predicting average arousal and valence up to an F1 score of 50.91% and 35.74% respectively.KeywordsMultimodal deep learningSentiment analysisFeature fusionLong short term memoryArousalValenceDeep learning