Efficient Multi-Lingual Sentence Classification Framework with Sentence Meta Encoders

Abstract

Natural Language Understanding (NLU) has become a primary paradigm in enterprise settings for myriad industrial applications like user intent classification, smarter chatbots, sentiment analysis, and duplicate detection to name a few. With the advent of globalization, significant advancements have been recently achieved in transformers-based multi-lingual language models such as XLM and its variants for downstream multi-lingual sentence or short text classification tasks. However, fine-tuning such large pre-trained language models is highly resource-intensive as it assumes the adaptation of the full model, hampering its wide adoption in production grade applications due to the demanding computational and memory requirements.In this paper, we present a practical and efficient framework based on fusing various pre-trained sentence encoders leveraging the multi-lingual knowledge distillation approach. We demon-strate, for the first time, the practicality of utilizing such multi-lingual sentence embeddings for supervised learning tasks with a focus on sentence classification scenarios. We experimented our proposed framework on a wide range of open source classification datasets and exhibit very competitive performance compared to fine-tuning large pre-trained language models. We showcase that our light-weight framework provides the advantage of ease of training within minutes on a single CPU, competitive inference time, and robustness to parameter settings. In hope of facilitating and democratizing practical research focused on NLP, we are planning to release our code as well as a new pre-trained sentence embeddings for XLM-R-large model.