Abstract Clinical management of IDH wildtype glioblastoma (GBM) and tumefactive multiple sclerosis (tMS) is drastically different. GBM requires maximal safe resection followed by chemoradiation, while tMS outcome is worsened by surgery and radiotherapy. Misdiagnosis of brain lesions may expose patients to unnecessary anxiety, surgery, or radiotherapy. Noninvasive methods to help with accurate diagnosis of tumor and non-tumor etiologies are needed. tMS subjects were identified by the Mayo Clinic Center for Multiple Sclerosis and Autoimmune Neurology (CMSAN). GBM subjects were matched to tMS by age at diagnosis, index MRI date, and sex. Inclusion criteria included one cm minimal lesion size and pre-operative post-contrast T1 and T2 images available for analysis. To identify potential experimental design bias, MRI technology and clinical characteristics were compared across groups. A 3D-DenseNet121 was used to develop a classification model using prespecified parameters: 600 epochs, batch size 16, learning rate 10-3, cross entropy loss, and AdamW optimizer. The stopping rule was defined as three sequential epoch cross entropy loss < 0.02. Five-fold cross validation (CV) was utilized to obtain an unbiased estimate of predictive performance (overfitting) and sensitivity of the 3D-DenseNet121 algorithm was evaluated. 224 subjects (114 GBM, 110 tMS) were analyzed: 40% male, median age 46. After one round of CV, the average unbiased estimate of sensitivity and specificity for predicting GBM from the final classification model was 0.86 and 0.73, respectively. To assess the reproducibility of the deep learning algorithm (3D-DenseNet121), the exact same CV folds were rerun resulting in average sensitivity and specificity of 0.82 and 0.75, respectively. Obtaining unbiased estimates of model performance using internal validation is critical before evaluating a final model on an external dataset. Understanding sensitivity of a deep learning algorithm is important for model validity. Differentiation of GBM and tMS appears reasonable with MRI; external validation is in progress.