A ranking prediction strategy assisted automatic model selection method

Abstract

With the development of booming AutoML systems, modeling processes have become more automatic for researchers. However, AutoML systems may struggle to identify the optimal surrogate type, find the best combination of the hyper-parameters or establish a high-fidelity ensembled surrogate model for certain datasets. To address these issues and further improve the warm-start procedure of AutoML, a Ranking Prediction Strategy assisted Automatic Model Selection (RPS-AMS) method is proposed. In the suggested method, an integration of evolutionary algorithms (EA-based) and feature-based driven model selection strategy selects the best or the best combination models for prediction. Based on the proposed criteria, an XGBoost regression model is trained to determine the rankings from the candidate surrogate models and then build an ensembled surrogate model to further enhance accuracy. We evaluate RPS-AMS using 13 mathematical functions, 14 public datasets, and a real engineering problem. Compared with the popular modeling tools, such as Auto-Sklearn and EvalML, the RPS-AMS outperforms in term of accuracy while maintaining the performances of ergodic methods of all surrogate models. The accuracies of RPS-AMS rival EvalML in most tested datasets, although RPS-AMS may be slightly less efficient. Given that EvalML is a masterpiece of the AutoML systems, the performances of RPS-AMS are promising. This code is available at: https://github.com/HnuAiSimOpt/RPS-AMS.