IcsBERTs: Optimizing Pre-trained Language Models in Intelligent Customer Service

Abstract

Automatic processing of textual information is a growing application area in intelligent customer service platforms due to the large number of customer requests constantly provided in the form of text. Many pre-trained language models have shown their high performance on text processing tasks. However, these pre-training strategies do not leverage domain-specific information. In this paper, we propose icsBERTs optimized for intelligent customer service on both word and sentence levels. Specifically, the automatic business words extraction and multi-task training methods are designed to enhance language representations for our models. Furthermore, we also propose the use of adaptors based on these icsBERTs for more complicated tasks. Our models outperform mainstream pre-trained language models on several tasks in the field of intelligent customer service and these experimental results demonstrate that using targeted strategies can further improve the performance of pre-trained language models in this field.