Real-time clickstream data analytics and visualization

Abstract

Nowadays web is becoming a main channel for reaching customers and prospects; Clickstream data generated by websites has become another important enterprise data source. As simple as it sounds for recording every click a customer made, so that we can use clickstream data for modelling user behavior, gaining valuable customer insights. Clickstream analysis commonly refers to analyzing click data and website optimization. Such analysis is typically done to extract insights into website visitor behavior especially social-media or e-commerce websites. Also nowadays online-learning became a trend in education system. We can see many online learning portals which are providing live training on various technologies. To identify potential customers or to identify recommendations for existing customers. Clickstream analysis can be used to figure out which geographies and time zones is most of traffic coming from, and which devices, Browsers (such as its name, versions), time spent, Operating Systems, are used to access the websites, which common paths users take before they do something in site. Analysis of clickstream data in real time(streaming) has more value than batch mode(stored). We analyze and visualize online-learning portal's clickstream data on the fly for business intelligence purpose. We constructed a full data pipeline using tools such as Apache Kafka, Apache Spark for streaming and elastic search, Kibana to query and visualize clickstream data respectively.