Sentiment Drift Detection and Analysis in Real Time Twitter Data Streams

Abstract

Handling sentiment drifts in real time twitter data streams are a challenging task while performing sentiment classifications, because of the changes that occur in the sentiments of twitter users, with respect to time. The growing volume of tweets with sentiment drifts has led to the need for devising an adaptive approach to detect and handle this drift in real time. This work proposes an adaptive learning algorithm-based framework, Twitter Sentiment Drift Analysis-Bidirectional Encoder Representations from Transformers (TSDA-BERT), which introduces a sentiment drift measure to detect drifts and a domain impact score to adaptively retrain the classification model with domain relevant data in real time. The framework also works on static data by converting them to data streams using the Kafka tool. The experiments conducted on real time and simulated tweets of sports, health care and financial topics show that the proposed system is able to detect sentiment drifts and maintain the performance of the classification model, with accuracies of 91%, 87% and 90%, respectively. Though the results have been provided only for a few topics, as a proof of concept, this framework can be applied to detect sentiment drifts and perform sentiment classification on real time data streams of any topic.