Kafka-ML: Connecting the data stream with ML/AI frameworks

Cristian Martín,Peter Langendoerfer,Pouya Soltani Zarrin,Manuel Díaz,Bartolomé Rubio

doi:10.1016/j.future.2021.07.037

Abstract

Machine Learning (ML) and Artificial Intelligence (AI) depend on data sources to train, improve, and make predictions through their algorithms. With the digital revolution and current paradigms like the Internet of Things, this information is turning from static data to continuous data streams. However, most of the ML/AI frameworks used nowadays are not fully prepared for this revolution. In this paper, we propose Kafka-ML, a novel and open-source framework that enables the management of ML/AI pipelines through data streams. Kafka-ML provides an accessible and user-friendly Web user interface where users can easily define ML models, to then train, evaluate, and deploy them for inferences. Kafka-ML itself and the components it deploys are fully managed through containerization technologies, which ensure their portability, easy distribution, and other features such as fault-tolerance and high availability. Finally, a novel approach has been introduced to manage and reuse data streams, which may eliminate the need for data storage or file systems.

Highlights

In this digital era, information is continuously acquired and processed everywhere, from many sources and for many purposes and sectors
Kafka-Machine Learning (ML) is characterized by its accessibility and ease of use since users need only a few lines of source code to create an ML model in its Web UI to control the ML/Artificial Intelligence (AI) pipeline, creating configurations to evaluate different ML models, training, validating, and deploying trained models for inference
Kafka-ML offers an innovative and opensource solution to manage the daily tasks performed by many ML/AI researchers and developers worldwide

Summary

Introduction

Information is continuously acquired and processed everywhere, from many sources and for many purposes and sectors. Companies like Facebook [2] process millions of photos every day to detect inappropriate contents This creates a continuous data stream for ML/AI algorithms and systems to face. With the rise of the Internet of Things (IoT) [3], new sources of data have been enabled in the Internet era, with a forecast of 500 billion connected devices by 2030 [4] Paradigms such as Industry 4.0, connected cars, and smart cities have become a possibility and, more importantly, they have contributed to the digitization of services in the physical world. In contrast to message queues, publish/subscribe systems allow multiple consumers to receive each message in a topic. In order to satisfy both requirements, Apache Kafka provides the following features:

Objectives

Methods

Results

Discussion

Conclusion