A multi-stage multimodal framework for sentiment analysis of Assamese in low resource setting

Abstract

Multimodality has shown to be helpful in several natural language processing tasks. Thus, adding multiple modalities to the traditional sentiment analysis has also proven to be useful. However, multimodality in a low resource setting for sentiment analysis is yet to be explored for several resource-constrained languages. Assamese is a low-resource language spoken mainly in the state of Assam in India. This paper presents an Assamese multimodal dataset comprising of 16,000 articles from the news domain as a benchmark resource. Secondly, we present a multi-stage multimodal sentiment analysis framework that concurrently exploits textual and visual cues to determine the sentiment. The proposed architecture encodes the news content collaboratively. The text branch encodes semantic content information by considering the semantic information of the news. At the same time, the visual branch encodes the visual appearance information from the news image. Then, an intermediate fusion-based multimodal framework is proposed to exploit the internal correlation between textual and visual features for joint sentiment classification. Finally, a decision-level fusion mechanism is employed on the three models to integrate cross-modal information effectively for final sentiment prediction. Experimental results conducted on the Assamese dataset built in-house demonstrate that the contextual integration of multimodal features delivers better performance (89.3%) than the best unimodal features (85.6%).