Distributed Machine Learning—An Intuitive Approach

Abstract

AbstractThe scope for automated systems by enabling artificial intelligence has grown rapidly over the last decade and this growth has been stimulated by advances in machine learning techniques by exploiting hardware acceleration. In order to improve the quality of predictions and utilize machine learning solutions feasible for more complex applications, a tangible amount of training data is necessary. Although relatively tiny machine learning models can be trained with modest amounts of data, the input for training larger models such as neural networks grows exponentially with the number of parameters. Since the demand for processing training data has outpaced the increase in computation power of computing machinery, there is a need for distributing the machine learning workload across different machines. That leads to a switch from the centralized into a distributed system. These distributed systems present new challenges; first and foremost, the efficient parallelism of the training process and the creation of a coherent model. This article provides an extensive overview of the current state-of-the-art in the field by outlining the challenges and opportunities of distributed machine learning over conventional (centralized) machine learning, discussing the techniques used for distributed machine learning, and providing an overview of the systems that are available.KeywordsClassical machine learningDistributed machine learning