Multi-task Learning with Bidirectional Language Models for Text Classification

Abstract

Multi-task learning is an effective approach to extract task-invariant features by leveraging potential information among related tasks, which improves the performance of a single task. Most existing work simply divides the whole model into shared and private spaces. Unfortunately, there is no explicit mechanism to prevent the two spaces from merging information from each other. As a result, the shared space may be mixed with task-specific features, while the private space may extract some task-invariant features. To alleviate the problem mentioned, in this paper, we propose a bidirectional language models based multi-task learning method for text classification. More specifically, we add language modelling as an auxiliary task to the private part, aiming to enhance its ability to extract task-specific features. In addition, to promote the shared part to learn common features, a loss constraint via uniform label distribution is introduced to the shared part. Finally, put task-specific features and taskinvariant features together in a weighted addition way to form the final representation, and it is then fed to the corresponding softmax layer. We do experiments on the FDU-MTL dataset which consists of 16 different text classification tasks. The experimental results show that our approach outperforms other typical methods.