HardGBM: A Framework for Accurate and Hardware-Efficient Gradient Boosting Machines

Abstract

Gradient boosting machine (GBM) is a powerful and widely used type of ensemble machine learning methods, among which the most famous one is XGBoost. However, the cost of running large GBMs on hardware could become prohibitive given stringent resources. Ensemble reduction on boosting ensemble is intrinsically hard because member models are constructed in a sequential order, where the training targets for latter ones depend on the performance of the former ones. In this work, a GBM reduction framework is proposed for the first time to tackle the problem. For the first time, the framework supports automatic hardware implementation of regression tree ensembles. Experiments on 24 datasets from various applications demonstrate that our method reduces overall area utilization by 81.60% (80.64%) and power consumption by 21.15% (19.06%), while exceeding or successfully maintaining the performance level comparing with the original XGBoost (LightGBM) ensembles. In comparative experiments, to attain approximately the same accuracy level as our framework or XGBoost, deep learning-based solutions require >52.7× footprints, 6.0× power consumption, and 1.4× training time. Equipped with tunable parameters, the framework is expected to seek a Pareto optimal front considering hardware resource limitation, accuracy and stability, and computation (training) efficiency.