Abstract

Semi-supervised learning through deep generative models and multi-lingual pretraining techniques have orchestrated tremendous success across different areas of NLP. Nonetheless, their development has happened in isolation, while the combination of both could potentially be effective for tackling task-specific labelled data shortage. To bridge this gap, we combine semi-supervised deep generative models and multi-lingual pretraining to form a pipeline for document classification task. Compared to strong supervised learning baselines, our semi-supervised classification framework is highly competitive and outperforms the state-of-the-art counterparts in low-resource settings across several languages.

Highlights

  • Multi-lingual pretraining has been shown to effectively use unlabelled data through learning shared representations across languages that can be transferred to downstream tasks (Artetxe and Schwenk, 2019; Devlin et al, 2019; Wu and Dredze, 2019; Conneau and Lample, 2019)

  • Combining with semisupervised DGMs (SDGMs), our best pipeline outperforms all baselines across data sizes and languages, including BERTW+ST with bigger gaps in fewer labelled data scenario

  • We observe the same trend of performance in both supervised and semisupervised deep generative models (DGMs) settings on EN-FR and DE-FR

Read more

Summary

Introduction

Multi-lingual pretraining has been shown to effectively use unlabelled data through learning shared representations across languages that can be transferred to downstream tasks (Artetxe and Schwenk, 2019; Devlin et al, 2019; Wu and Dredze, 2019; Conneau and Lample, 2019). Deep generative models (DGMs) such as variational autoencoder (VAE; Kingma and Welling (2014)) are capable of capturing complex data distributions at scale with rich latent representations, and they have been used for semi-supervised learning in various tasks in NLP (Xu et al, 2017; Yin et al, 2018; Choi et al, 2019; Xie and Ma, 2019), as well as inducing crosslingual word embeddings (Wei and Deng, 2017), and representation learning in combination with Transformers via pretraining (Li et al, 2020). The pretrained model serves as multilingual encoder, and SDGMs can operate on top of it independently of encoding architecture To highlight such independence, we experiment with two pretraining settings: (1) our LSTM-based crosslingual VAE, and (2) the current stat-of-the-art (SOTA) multi-lingual BERT (Devlin et al, 2019)

Objectives
Methods
Results
Conclusion

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.