Formalstyler: GPT-Based Model for Formal Style Transfer with Meaning Preservation

Abstract

Style transfer is a natural language processing generation task, it consists of substituting one given writing style for another one. In this work, we seek to perform informal-to-formal style transfers in the English language by using a style transfer model that takes advantage of the GPT-2. This process is shown in our web interface where the user input a informal message by text or voice. Our target audience are students and professionals in the need to improve the quality of their work by formalizing their texts. A style transfer is considered successful when the original semantic meaning of the message is preserved after the independent style has been replaced with a formal one with a high degree of grammatical correctness. This task is hindered by the scarcity of training and evaluation datasets alongside the lack of metrics. To accomplish this task, we opted to utilize OpenAI’s GPT-2 Transformer-based pre-trained model. To adapt the GPT-2 to our research, we fine-tuned the model with a parallel corpus containing informal text entries paired with the equivalent formal ones. We evaluate the fine-tuned model results with two specific metrics, formality and meaning preservation. To further fine-tune the model, we integrate a human-based feedback system where the user selects the best formal sentence out of the ones generated by the model. The resulting evaluations of our solution exhibit similar to improved scores in formality and meaning preservation to state-of-the-art approaches.