Predicting and Exploring Abandonment Signals in a Banking Task-Oriented Chatbot Service

Abstract

In this study, we developed predictive models to address the problem of chatbot abandonment, a problem that can result in losing business opportunities. Specifically, we target on a conversation log dataset of a banking chatbot involving 1,373 users and hand-crafted features. By leveraging a pre-trained BERT model on the textural features and the hand-crafted features, the model achieved an F1-score of 0.89 in predicting discontinued conversation and 0.80 in predicting abandonment. Our findings indicate that textual features help capture more abandonment, while hand-crafted features improve detection precision. Our analysis with SHAP and LIME revealed that user typing, the chatbot expressing inability of recognizing intent, and the chatbot asking what users want to do during an ongoing conversation are top signals of user abandoning the chatbot. These findings suggest that chatbot designers should consider providing pre-set options or constraints for user inputs and presenting possible intents to the user and avoid expressing inability, incompetence, or ignoring the users’ current attempt.