Neural Transfer Learning For Vietnamese Sentiment Analysis Using Pre-trained Contextual Language Models

Abstract

Listen

Translate article

We propose a fine-tuning methodology and a comprehensive comparison between state-of-the-art pre-trained language models (PLM) when applying to Vietnamese Sentiment Analysis. The fine-tuning architecture includes three main components: (1) pre-processing, (2) a pre- trained language model, and (3) a multi-layer perceptron. The method exploits pre-trained contextual language models in order to represent input sentences. Pre-trained contextual language models are belong to three different kinds: multilingual, cross-lingual and monolingual. We conduct experiments to evaluate trained classifiers fine-tuned using five different contextual language models. The experimental results on two open-access datasets show that the sentiment classifiers trained using the monolingual language model outperform of which cross-lingual and monolingual language models. The results provide an additional evidence about the representation power of monolingual PhoBERT in comparison with multilingual BERT and cross-lingual XLM.