Towards flexible data stream collaboration: Federated Learning in Kafka-ML

Abstract

Federated learning is applied in scenarios where organisations lack sufficient data volume for modelling their business logic and cannot share their data with external parties. Moreover, Industry 4.0 and IoT scenarios generate massive data streams, which normally are fed to ML/AI solutions for model training and prediction. However, in most cases, ML/AI frameworks are not prepared to work with these streaming pipelines. In this paper, we present an asynchronous federated learning solution based on the Kafka-ML data stream framework, which is able to combine federated learning and data stream capabilities within ML/AI applications. While most federated learning approaches are tailored to a specific ML model or a use case, the solution provided adapts itself to the availability of both data and ML models, achieving a flexible and dynamic federated learning solution. To validate its performance, an evaluation of the federated learning solution is carried out on different scenarios in a multi-node state-of-the-art infrastructure. Results show that this framework can work with multiple federated clients, being the resulting accuracy dependent on the amount of data and the behaviour of clients during training.