Few-Shot Multi-Modal Sentiment Analysis with Prompt-Based Vision-Aware Language Modeling

Abstract

As a hot study topic in natural language processing, affec-tive computing and multimedia analysis, multi-modal senti-ment analysis (MSA) is widely explored on aspect-level and sentence-level tasks. However, the existing studies normally rely on a lot of annotated multi-modal data, which are difficult to collect due to the massive expenditure of manpower and re-sources, especially in some open-ended and fine-grained do-mains. Therefore, it is necessary to investigate the few-shot scenario for MSA. In this paper, we propose a prompt-based vision-aware language modeling (PVLM) approach to MSA, which only requires a few supervised data. Specifically, our PVLM can incorporate the visual information into pre-trained language model and leverage prompt tuning to bridge the gap between masked language prediction in pre-training and MSA tasks. Systematic experiments on three aspect-level and two sentence-level datasets of MSA demonstrate the effectiveness of our few-shot approach.