Bias-variance tradeoff in machine learning: Theoretical formulation and implications to structural engineering applications

Abstract

When machine learning (ML) is used to solve practical problems in structural engineering (SE), the associated data may not reflect the overall distribution of the population. With this in mind, a common practice in prior ML-SE studies is to optimize the model such that the testing error is minimized. However, this strategy can lead to a situation where the testing error is significantly greater than the error obtained during training, which is a clear sign of overfitting. Therefore, simply selecting a model with the highest testing performance does not necessarily guarantee generalizability. To address this challenge, this study aims to provide some practical guidelines that consider both the model complexity and generalizability. The proposed solution is fairly straightforward to implement. The generalized bias-variance decomposition is first introduced, followed by a discussion of different error variation patterns (bell-shaped, double-descent, and monotonically decreasing). A mathematical formulation that establishes a relationship between the model complexity and generalizability via an objective function is then presented to facilitate the model parameter selection. Subsequently, four numerical experiments are conducted to apply the formulation within the structural engineering domain. Finally, the proposed model selection approach is compared with existing methods to demonstrate its advantages and limitations.