Efficient dynamic feature adaptation for cross language sentiment analysis with biased adversarial training

Abstract

Fine-tuning a large multi-lingual pretrained language model demonstrates impressive results in cross-language understanding. However, it still suffers when the training and test data have different distributions owing to various languages and domains. On one hand, annotating target data for different languages or domains is time-consuming or infeasible. On the other hand, fine-tuning a large language model often incurs high computational costs. In this paper, we aim to develop an efficient and effective adaptation framework for cross-language sentiment analysis based on a fixed pretrained multi-lingual model. Specifically, we propose a Dynamic Feature Adaptation (DFA) module to fully leverage the features from different layers of the pretrained model such that its large backbone is not involved during adaptation training. Furthermore, we observe that traditional adversarial domain adaptation training could compromise the discriminative information of the model by pushing source and target features towards each other. The source features obtained with supervised training preserved the discriminability of the model, which should be less affected. Therefore, we propose a novel Biased Adversarial Training (BAT) method, that encourages only the target features towards source features. Extensive experimental results on various cross-lingual and cross-lingual-and-domain sentiment analysis tasks demonstrate the superiority of the proposed framework. Additionally, several ablation studies are conducted to validate the effectiveness of each proposed module.