Altering the Decision Threshold as a Simple and Effective Method for Machine Learning-Based Classification of Imbalanced Radiation Oncology Data

Abstract

Imbalanced outcomes are ubiquitous in radiation oncology, such as high local control rate for lung SBRT, low two-year survival for glioblastoma, or low rate of grade 4-5 complications. Machine learning classifiers traditionally require balanced training data. Two balancing approaches are undersampling the majority class (reduces the training set size), or generating synthetic samples of the minority class (may alter the true correlation between a feature and the outcome). We hypothesize that training on the imbalanced set and optimizing the decision threshold of the classifier can be a superior alternative. The dataset comprised 525 contrast-enhanced mammograms (250 benign, 275 malignant) and 3259 derived radiomic features. It was randomly partitioned 10 times into discovery (n = 450) and test sets (n = 50) that were balanced relative to malignancy. From each discovery set, imbalanced training sets (n = 250) were randomly sampled using nine imbalance levels (ILs) by varying the benign class fraction F from 0.1−0.9 (25:225, 50:200, 75:175,…). Each IL was instantiated 10 times per discovery set yielding 100 training sets per IL. Features with Area Under the Curve (AUC) > 0.8 were retained for each training set. Earlier work has shown that AUC is unaffected by data imbalance, unlike correlation coefficients. Pairwise feature elimination was subsequently performed: if two features were highly correlated with each other (|R|>0.7 for linear classifiers, and |R|>0.9 for Random Forest), the one with lower AUC was removed. Logistic regression (LR), Naïve Bayes, and linear discriminant were used as linear classifiers. The optimized decision threshold (ODT) was defined as the threshold corresponding to the point on the Receiver Operating Characteristic (ROC) curve nearest to the point denoting a false-positive-rate of 0 and a true-positive-rate of 1. For all classifiers, the ODT was the prior probability of the benign class, except for LR, where the difference between the model intercept and the ODT was a fixed value. Thus, for all cases, the ODT could be obtained without the ROC curve. The table shows the comma-separated training and test accuracies for various Fs (each accuracy is a mean of 100 values). By using the ODT, all models achieved balanced sensitivity and specificity (mean values within 5% in all cases), leading to excellent agreement between training and test performance. Our method successfully trains classifiers on imbalanced sets by altering the decision threshold. It does not suffer from the limitations of undersampling or synthetic oversampling, nor does it increase computation time. Validation on other clinical datasets is needed.Tabled 1Abstract 2748; TableF = 0.1F = 0.3F = 0.5F = 0.7F = 0.9Logistic Regression0.90, 0.850.86, 0.860.86, 0.860.86, 0.850.86, 0.81Naïve Bayes0.85, 0.840.85, 0.840.85, 0.830.84, 0.830.85, 0.83Linear Discriminant0.88, 0.850.85, 0.840.86, 0.850.86, 0.840.86, 0.83Random Forest0.93, 0.880.90, 0.870.90, 0.870.91, 0.850.92, 0.83 Open table in a new tab