Benchmarking AutoML solutions for concrete strength prediction: Reliability, uncertainty, and dilemma

Abstract

Building precise machine learning and deep learning models has traditionally required a combination of mathematical skills and hands-on experience to meticulously adjust hyperparameters that significantly impact the learning process. As datasets continue to expand across various engineering domains, researchers increasingly turn to machine learning methods to uncover hidden insights that may elude classic regression techniques. This surge in adoption raises concerns about the adequacy of resultant meta-models and the interpretation of findings. In response to these challenges, automated machine learning (AutoML) emerges as a promising solution, aiming to construct machine learning models with minimal intervention or guidance from human experts. This paper benchmarks AutoML solutions by providing an overview of their principles and applying them to predict the most important mechanical properties of different concrete datasets, i.e., compressive strength. Nine datasets from various concrete types, sample sizes, and features are utilized, with a detailed discussion on the benchmark dataset from high-performance concrete, applying best practices to the other eight datasets. For each case, the importance of hyperparameter tuning is discussed, alongside the ensemble and stacking models. Tree-based models are employed for each dataset to develop SHAP plots, interpret results, and understand the contribution of each component in the mix design to the overall strength of the concrete. This paper further explores three unique aspects of benchmarking AutoML in material science: (1) “reliability” by contrasting the benchmark dataset’s error metric with literature collected over the past 20 years, (2) “uncertainty” by quantifying the variability in the mean and standard deviation of the error metric from different datasets and its correlation with the sample-to-feature ratio, and (3) “dilemma” by discussing the shortcomings of AutoML in specific concrete datasets.