Pseudo-labeling with transformers for improving Question Answering systems

Abstract

Advances in neural networks contributed to the fast development of Natural Language Processing systems. As a result, Question Answering systems have evolved and can classify and answer questions in an intuitive yet communicative way. However, the lack of large volumes of labeled data prevents large-scale training and development of Question Answering systems, confirming the need for further research. This paper aims to handle this real-world problem of lack of labeled datasets by applying a pseudo-labeling technique relying on a neural network transformer model DistilBERT. In order to evaluate our contribution, we examined the performance of a text classification transformer model that was fine-tuned on the data subject to prior pseudo-labeling. Research has shown the usefulness of the applied pseudo-labeling technique on a neural network text classification transformer model DistilBERT. The results of our analysis indicated that the model with additional pseudo-labeled data achieved the best results among other compared neural network architectures. Based on that result, Question Answering systems may be directly improved by enriching their training steps with additional data acquired cost-effectively.