Automatic update strategy for real-time discovery of hidden customer intents in chatbot systems

Abstract

Model degradation is still a challenge in real-time applications such as chatbot systems. This work refers to a webchat service of a Brazilian energy utility company, whose central part is composed of a supervised model and a question-and-answer list. User queries not met by it go to an NLP-based clustering model, responsible for identifying unknown customer intents. Manual labeling is impractical in this case due to the large volume of data. This work proposes an automatic update strategy for this clustering model, necessary due to changes in customer behavior from time to time. A series of experiments showed considerable temporal variation in the number of user queries per customer intent, the allocation of queries from unknown intents to few clusters, and larger relative variations in cluster sizes for unknown rather than known intents over time. Based on these findings, a monitoring metric, together with a cut-off point, was proposed to be used as a trigger for updating the clustering model. This update task was demonstrated in a real situation, from which the discovery of new customer intents was confirmed by experts. This resulted in a significant recovery rate of 13.9% (2251 messages), as 85.1% of the queries are already answered promptly by the central part of the chatbot system. These findings are valuable for the company to improve service quality and, ultimately, customer satisfaction.