Scalable and real-time prediction on streaming data - the role of Kafka and streaming frameworks

Abstract

Increasingly data gets generated and streamed from different and distributed data sources. In order to make sense out of any streaming data, the market is flooded with a variety of features-rich streaming data analytics platforms and frameworks. Lately, the role and responsibility of Kafka, an open source streaming data platform, are growing steadily. In this chapter, we have written about the noteworthy contributions of Kafka in producing timely, trendsetting and predictive insights out of streaming data.A new breed of "Fast Data" architectures has evolved to be stream-oriented, where data is processed as it arrives, providing businesses with a competitive advantage. The demand for stream processing is increasing every day in the digital era. The main reason behind it is that processing only volumes of data is not sufficient but processing data at faster rates and making insights out of it in real time is very essential so that organization can react to changing business conditions/sentiments in real time. And hence, there is a need to understand the concept "stream processing" and technology behind it. This collateral is prepared with the noble intention of articulating the need for scalable and real-time prediction on streaming data. There are competent technologies, tools, and techniques for real-time prediction out of streaming data in a highly elastic manner. All these details are covered in this document in order to enlighten our readers. The major topics illustrated here include; 1. Streaming concepts; 2. Apache Kafka; 3. Apache Kafka streams; 4. Apache Spark; 5. A sample machine learning (ML) application for real-time prediction out of streaming data in a scalable fashion.