Multi-level multilingual semantic alignment for zero-shot cross-lingual transfer learning

Abstract

Recently, cross-lingual transfer learning has attracted extensive attention from both academia and industry. Previous studies usually focus only on the single-level alignment (e.g., word-level, sentence-level), based on pre-trained language models. However, it leads to suboptimal performance in downstream tasks of the low-resource language due to the missing correlation of hierarchical semantic information (e.g., sentence-to-word, word-to-word). Therefore, in this paper, we propose a novel multi-level alignment framework, which hierarchically learns the semantic correlation between multiple levels by leveraging well-designed alignment training tasks. In addition, we devise an attention-based fusion mechanism (AFM) to infuse semantic information from high levels. Extensive experiments on mainstream cross-lingual tasks (e.g., text classification, paraphrase identification, and named entity recognition) demonstrate the effectiveness of our proposed method, and also show that our model achieves state-of-the-art performance across various benchmarks compared to other strong baselines.