Foundations of Machine Learning-Based Clinical Prediction Modeling: Part III-Model Evaluation and Other Points of Significance.

Abstract

Various available metrics to describe model performance in terms of discrimination (area under the curve (AUC), accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1 Score) and calibration (slope, intercept, Brier score, expected/observed ratio, Estimated Calibration Index, Hosmer-Lemeshow goodness-of-fit) are presented. Recalibration is introduced, with Platt scaling and Isotonic regression as proposed methods. We also discuss considerations regarding the sample size required for optimal training of clinical prediction models-explaining why low sample sizes lead to unstable models, and offering the common rule of thumb of at least ten patients per class per input feature, as well as some more nuanced approaches. Missing data treatment and model-based imputation instead of mean, mode, or median imputation is also discussed. We explain how data standardization is important in pre-processing, and how it can be achieved using, e.g. centering and scaling. One-hot encoding is discussed-categorical features with more than two levels must be encoded as multiple features to avoid wrong assumptions. Regarding binary classification models, we discuss how to select a sensible predicted probability cutoff for binary classification using the closest-to-(0,1)-criterion based on AUC or based on the clinical question (rule-in or rule-out). Extrapolation is also discussed.