Real-Time Big Data Analysis Using Web Scraping in Apache Spark Environment: Case Study—Mobile Data Analysis from Flipkart

Abstract

AbstractThere are many data sources that produce sheer volumes of data, and this generated data named big data can be used for making decisions. There are lots of domains including finance, transportation, entertainment, energy, security, and emergency services rely on fast and efficient analytics based on available data to make quality decisions conveniently which is a key factor for businesses and many service industries. The big data nature requires new distributed processing approaches to extract the valuable information. Real-time sentiment analysis is one of the most demanding research areas that require powerful big data analytics tools such as Spark. This paper proposed a real-time data analysis. It first scraps data from the Web and will do real-time analysis using Spark with machine learning tools. As a case study for this real-time analysis, Flipkart mobile data is taken with the aim of doing classification of the dataset for further analysis as well as predicting the ratings of different kinds of mobile phones using machine learning modules. At last the accuracy of the implemented models is shown.KeywordsData streamingReal-time data analysisExploratory data analysisMachine learning modelsData scraping