Abstract
Machine Learning (ML) and Artificial Intelligence (AI) depend on data sources to train, improve, and make predictions through their algorithms. With the digital revolution and current paradigms like the Internet of Things, this information is turning from static data to continuous data streams. However, most of the ML/AI frameworks used nowadays are not fully prepared for this revolution. In this paper, we propose Kafka-ML, a novel and open-source framework that enables the management of ML/AI pipelines through data streams. Kafka-ML provides an accessible and user-friendly Web user interface where users can easily define ML models, to then train, evaluate, and deploy them for inferences. Kafka-ML itself and the components it deploys are fully managed through containerization technologies, which ensure their portability, easy distribution, and other features such as fault-tolerance and high availability. Finally, a novel approach has been introduced to manage and reuse data streams, which may eliminate the need for data storage or file systems.
Highlights
In this digital era, information is continuously acquired and processed everywhere, from many sources and for many purposes and sectors
Kafka-Machine Learning (ML) is characterized by its accessibility and ease of use since users need only a few lines of source code to create an ML model in its Web UI to control the ML/Artificial Intelligence (AI) pipeline, creating configurations to evaluate different ML models, training, validating, and deploying trained models for inference
Kafka-ML offers an innovative and opensource solution to manage the daily tasks performed by many ML/AI researchers and developers worldwide
Summary
Information is continuously acquired and processed everywhere, from many sources and for many purposes and sectors. Companies like Facebook [2] process millions of photos every day to detect inappropriate contents This creates a continuous data stream for ML/AI algorithms and systems to face. With the rise of the Internet of Things (IoT) [3], new sources of data have been enabled in the Internet era, with a forecast of 500 billion connected devices by 2030 [4] Paradigms such as Industry 4.0, connected cars, and smart cities have become a possibility and, more importantly, they have contributed to the digitization of services in the physical world. In contrast to message queues, publish/subscribe systems allow multiple consumers to receive each message in a topic. In order to satisfy both requirements, Apache Kafka provides the following features:
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
