Using VADER sentiment and SVM for predicting customer response sentiment

Abstract

Customer support is important to corporate operations, which involves dealing with disgruntled customer and content customers that can have different requirements. As such, it is important to quickly extract the sentiment of support errands. In this study we investigate sentiment analysis in customer support for a large Swedish Telecom corporation. The data set consists of 168,010 e-mails divided into 69,900 conversation threads without any sentiment information available. Therefore, VADER sentiment is used together with a Swedish sentiment lexicon in order to provide initial labeling of the e-mails. The e-mail content and sentiment labels are then used to train two Support Vector Machine models in extracting/classifying the sentiment of e-mails. Further, the ability to predict sentiment of not-yet-seen e-mail responses is investigated. Experimental results show that the LinearSVM model was able to extract sentiment with a mean F1-score of 0.834 and mean AUC of 0.896. Moreover, the LinearSVM algorithm was also able to predict the sentiment of an e-mail one step ahead in the thread (based on the text in the an already sent e-mail) with a mean F1-score of 0.688 and the mean AUC of 0.805. The results indicate a predictable pattern in e-mail conversation that enables predicting the sentiment of a not-yet-seen e-mail. This can be used e.g. to prepare particular actions for customers that are likely to have a negative response. It can also provide feedback on possible sentiment reactions to customer support e-mails.