Large-scale distributed L-BFGS

Maryam M Najafabadi,Taghi M Khoshgoftaar,Flavio Villanustre,John Holt

doi:10.1186/s40537-017-0084-5

Abstract

With the increasing demand for examining and extracting patterns from massive amounts of data, it is critical to be able to train large models to fulfill the needs that recent advances in the machine learning area create. L-BFGS (Limited-memory Broyden Fletcher Goldfarb Shanno) is a numeric optimization method that has been effectively used for parameter estimation to train various machine learning models. As the number of parameters increase, implementing this algorithm on one single machine can be insufficient, due to the limited number of computational resources available. In this paper, we present a parallelized implementation of the L-BFGS algorithm on a distributed system which includes a cluster of commodity computing machines. We use open source HPCC Systems (High-Performance Computing Cluster) platform as the underlying distributed system to implement the L-BFGS algorithm. We initially provide an overview of the HPCC Systems framework and how it allows for the parallel and distributed computations important for Big Data analytics and, subsequently, we explain our implementation of the L-BFGS algorithm on this platform. Our experimental results show that our large-scale implementation of the L-BFGS algorithm can easily scale from training models with millions of parameters to models with billions of parameters by simply increasing the number of commodity computational nodes.

Highlights

A wide range of machine learning algorithms use optimization methods to train the model parameters [1]
In this paper, we explained a parallelized distributed implementation of L-BFGS which works for training large-scale models with billions of parameters
The L-BFGS algorithm is an effective parameter optimization method which can be used for parameter estimation for various machine learning problems

Summary

Introduction

A wide range of machine learning algorithms use optimization methods to train the model parameters [1]. The HPCC Systems platform provides a framework for a general solution for large-scale processing which is not limited to a specific implementation It allows manipulation of the data locally on each node (similar to the parameter server in [4]). Several organizations developed new technologies which utilize large clusters of commodity servers to provide the underlying platform to process and analyze massive data Some of these technologies include MapReduce [23,24,25], Hadoop [16] and the open source HPCC Systems. The data refinery cluster in HPCC Systems (Thor system cluster) is designed for processing massive volumes of raw data which ranges from data cleansing and ETL processing to developing machine learning algorithms and building large-scale models It functions as a distributed file system with parallel processing power spread across the nodes (machines). The section covers our implementation of this algorithm on HPCC Systems platform

7: Calculate αk where it satisfies Wolfe conditions

Results

# Objective function

Conclusion and discussion