Abstract
With the increasing demand for examining and extracting patterns from massive amounts of data, it is critical to be able to train large models to fulfill the needs that recent advances in the machine learning area create. L-BFGS (Limited-memory Broyden Fletcher Goldfarb Shanno) is a numeric optimization method that has been effectively used for parameter estimation to train various machine learning models. As the number of parameters increase, implementing this algorithm on one single machine can be insufficient, due to the limited number of computational resources available. In this paper, we present a parallelized implementation of the L-BFGS algorithm on a distributed system which includes a cluster of commodity computing machines. We use open source HPCC Systems (High-Performance Computing Cluster) platform as the underlying distributed system to implement the L-BFGS algorithm. We initially provide an overview of the HPCC Systems framework and how it allows for the parallel and distributed computations important for Big Data analytics and, subsequently, we explain our implementation of the L-BFGS algorithm on this platform. Our experimental results show that our large-scale implementation of the L-BFGS algorithm can easily scale from training models with millions of parameters to models with billions of parameters by simply increasing the number of commodity computational nodes.
Highlights
A wide range of machine learning algorithms use optimization methods to train the model parameters [1]
In this paper, we explained a parallelized distributed implementation of L-BFGS which works for training large-scale models with billions of parameters
The L-BFGS algorithm is an effective parameter optimization method which can be used for parameter estimation for various machine learning problems
Summary
A wide range of machine learning algorithms use optimization methods to train the model parameters [1]. The HPCC Systems platform provides a framework for a general solution for large-scale processing which is not limited to a specific implementation It allows manipulation of the data locally on each node (similar to the parameter server in [4]). Several organizations developed new technologies which utilize large clusters of commodity servers to provide the underlying platform to process and analyze massive data Some of these technologies include MapReduce [23,24,25], Hadoop [16] and the open source HPCC Systems. The data refinery cluster in HPCC Systems (Thor system cluster) is designed for processing massive volumes of raw data which ranges from data cleansing and ETL processing to developing machine learning algorithms and building large-scale models It functions as a distributed file system with parallel processing power spread across the nodes (machines). The section covers our implementation of this algorithm on HPCC Systems platform
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
