Primary Central Nervous System tumors in the brain are among the most aggressive diseases affecting humans. Early detection and classification of brain tumor types, whether benign or malignant, glial or non-glial, is critical for cancer prevention and treatment, ultimately improving human life expectancy. Magnetic Resonance Imaging (MRI) is the most effective technique for brain tumor detection, generating comprehensive brain scans. However, human examination can be error-prone and inefficient due to the complexity, size, and location variability of brain tumors. Recently, automated classification techniques using machine learning methods, such as Convolutional Neural Networks (CNNs), have demonstrated significantly higher accuracy than manual screening. However, deep learning-based image classification methods, including CNNs, face challenges in estimating class probabilities without proper model calibration (Guo et al., 2017; Minderer et al., 2021). In this paper, we propose a novel brain tumor image classification method called SIBOW-SVM, which integrates the Bag-of-Features model with SIFT feature extraction and weighted Support Vector Machines. This new approach can effectively extract hidden image features, enabling differentiation of various tumor types, provide accurate label predictions, and estimate probabilities of images belonging to each class, offering high-confidence classification decisions. We have also developed scalable and parallelable algorithms to facilitate the practical implementation of SIBOW-SVM for massive image datasets. To benchmark our method, we apply SIBOW-SVM to a public dataset of brain tumor MRI images containing four classes: glioma, meningioma, pituitary, and normal. Our results demonstrate that the new method outperforms state-of-the-art techniques, including CNNs, in terms of uncertianty quantification, classification accuracy, computational efficiency, and data robustness.