Bridging modalities: a deep learning framework for brain tumor classification via CT-MRI integration and model fusion.

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. The most important question when using multimodal data is how to fuse them—a field of growing interest among researchers. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. After pre-processing and screening, we extracted data from 34 studies that fulfilled the inclusion criteria. We found that studies fusing imaging data with EHR are increasing and doubling from 2020 to 2021. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective. Neurological disorders were the dominant category (16 studies). From an AI perspective, conventional ML models were the most used (19 studies), followed by DL models (16 studies). Multimodal data used in the included studies were mostly from private repositories (21 studies). Through this scoping review, we offer new insights for researchers interested in knowing the current state of knowledge within this research field.

Focus issue: Artificial intelligence in medical physics.

An optimized framework for Parkinson's disease classification using multimodal neuroimaging data with ensemble-based and data fusion networks.

BackgroundBrain tumor classification using Magnetic Resonance Imaging (MRI) is crucial for diagnosis and treatment planning. The differentiation between malignant and benign brain tumors and their subtypes remains a challenging task that can benefit from advanced computational techniques.PurposeThis study uses an MRI dataset to explore the effectiveness of deep learning (DL) and machine learning (ML) approaches for classifying brain tumors.Materials and methodsA dataset comprising 1200 DICOM brain tumor MRI images, representing malignant and benign tumors with six subtypes, was prepared. Each image was converted to a 512 × 512-pixel digital format, selecting 200 images per tumor class. Image quality was enhanced using sharpening algorithms and mean filtering. The proposed edge refined binary histogram segmentation (ER-BHS) was applied to extract hybrid features from the regions of interest. Feature optimization through a correlation-based method reduced the dataset to 11 key features. Multiple classifiers, including DL, neural networks, and ML models, were evaluated on the optimized dataset using 10-fold cross-validation.ResultsAmong the tested models, the random committee (RC) classifier demonstrated superior performance, achieving an accuracy of 98.61% on the optimized hybrid brain tumor MRI dataset. Overall, DL and ML methods effectively automated brain tumor classification.ConclusionThe promising results affirm the potential of DL and ML approaches to enhance medical image analysis and improve diagnostic accuracy in brain tumor classification, potentially revolutionizing clinical workflows.

Chapter 12 - A fine-tuned deep transfer learning model in classifying multiclass brain tumors for preclinical MRI image analysis

A brain tumor is one of the most perilous diseases in human beings. The manual segmentation of brain tumors is costly and takes a lot of time; due to this reason, automated approaches are highly valued. Automated brain tumor detection by radiologists is a difficult task for the surveillance of patients. Early detection of brain tumors has a significant role in enhancing the effectiveness of treatment and further increasing the survival of patients. Brain tumor detection is a challenging task for radiologists and physicians. It is difficult to examine the brain tumor through image processing generated in the medical. Thus, for early brain tumor detection, there is a critical requirement for computer-aided approaches with higher accuracy. Multi-model images nowadays are increasing the interest in the classification of brain tumors. The detection of brain tumors through magnetic resonance images (MRIs) consists of segmentation and classification methods. Many experiments have been done over the last few years on machine learning (ML) for brain tumor segmentation and classification. Recently, interest has increased in the use of deep learning (DL) approaches to help the detection of brain tumors. MRI is commonly used to detect the brain tissues according to size, shape or location, and that aids to detect the tumor. The main goal of this chapter is to help researchers extract the essential features of brain tumors detection and identify the different automated segmentation and classification techniques that are successful in using multimodal MRIs to detect brain tumors. Additionally, this chapter provides a comprehensive review and comparative study of various automated brain tumor classification methods/techniques built on ML and DL methods of brain tumor detection from MRI. The different strategies of ML and DL, including K-means clustering, fuzzy C-means, K-nearest neighbor, support vector machine, decision tree, G-convolutional neural network, artificial neural network (ANN), conditional random field-recurrent neural network (CRF-RNN), deep neural network (DNN), Naive Bayes, etc. are used in this work. Some famous methods are generally used to create valuable data from medical image processing techniques. The key achievements represented the algorithms performance measurement metrics that are identified in this chapter. The proposed methods by researchers are considered for the Medical Image Computing and Computer-Assisted Intervention (MICCAI) challenges on benchmark brain tumor segmentation (BRATS) 2012–2019, and many other different datasets for segmentation and classification of brain tumors are used in this chapter. The researchers have used several techniques for the segmentation, classification and detection of a brain tumor on various datasets for achieving the best performance. Automatic brain tumor detection through MRIs is essential, as high accuracy is required when working with human life. This study helps the researchers to choose the best method/technique with the use of different datasets.

Accurate and automated classification of brain tumors from magnetic resonance imaging (MRI) scans is essential for improving diagnostic precision and supporting clinical decision-making. This study presents a deep learning-based framework that employs two convolutional neural network architectures a custom-designed CNN and a pretrained ResNet18 model for multi-class classification of brain tumors using the publicly available Kaggle MRI dataset. The dataset was preprocessed through normalization, augmentation, and resizing to ensure consistency and model generalization. Both models were trained and evaluated using an 80:20 data split, and their performance was assessed based on accuracy, precision, recall, and F1-score metrics. Experimental results demonstrate that the ResNet18 model outperforms the baseline CNN, achieving a classification accuracy of 99.7%, precision of 99.5%, and F1-score of 99.6%. These results highlight the effectiveness of transfer learning and residual connections in improving feature representation and convergence speed. These findings underscore the effectiveness of transfer learning for medical image analysis and demonstrate the potential of deep learning"based methods for reliable, automated brain tumor diagnosis. Future research should focus on extending this work to 3D MRI volumes and integrating explainable AI techniques to enhance interpretability and clinical trust. Accurate classification of brain tumors from Magnetic Resonance Imaging (MRI) is crucial for early diagnosis, treatment planning, and improving patient outcomes. However, manual interpretation of MRI scans is time-consuming and susceptible to diagnostic inconsistencies. This study presents a comparative evaluation of a custom Convolutional Neural Network (CNN) and a transfer learning"based ResNet18 model for automated brain tumor classification using the Kaggle Brain Tumor MRI Dataset, which includes four diagnostic categories: glioma, meningioma, pituitary tumor, and no tumor. Both models were trained and validated under identical preprocessing and experimental conditions to ensure fair comparison. Comprehensive preprocessing, including normalization, augmentation, and stratified splitting (70% training, 20% validation, 10% testing), was applied to enhance data uniformity and generalization. The CNN model was trained from scratch, whereas the ResNet18 model is fine-tuned using pretrained ImageNet weights to leverage transfer learning. Performance was evaluated using accuracy, precision, recall, F1-score, and AUC metrics, supplemented by visual diagnostics such as confusion matrices, accuracy"loss curves, and F1-confidence plots. The ResNet18 model achieved superior performance, with a test accuracy of 99.54%, precision of 0.98, recall of 0.99, F1-score of 0.99, and AUC of 0.992, outperforming the custom CNN, which attained 97.84% accuracy, precision of 0.94, recall of 0.95, F1-score of 0.94, and AUC of 0.975. Confusion matrix analysis indicated that both models accurately classified all tumor types, though minor misclassifications were observed between pituitary and no-tumor categories. ResNet18 exhibited faster convergence, smoother loss reduction, and greater robustness to intensity variations due to its residual connections and pretrained feature representations.

Comprehensive Study for Breast Cancer Using Deep Learning and Traditional Machine Learning

Brain tumour detection and classification require trained radiologists for efficient diagnosis. The proposed work aims to build a Computer Aided Diagnosis (CAD) tool to automate brain tumour detection using Machine Learning (ML) and Deep Learning (DL) techniques. Magnetic Resonance Image (MRI) collected from the publicly available Kaggle dataset is used for brain tumour detection and classification. Deep features extracted from the global pooling layer of Pretrained Resnet18 network are classified using 3 different ML Classifiers, such as Support vector Machine (SVM), K-Nearest Neighbour (KNN), and Decision Tree (DT). The above classifiers are further hyperparameter optimised using Bayesian Algorithm (BA) to enhance the performance. Fusion of features extracted from shallow and deep layers of the pretrained Resnet18 network followed by BA-optimised ML classifiers is further used to enhance the detection and classification performance. The confusion matrix derived from the classifier model is used to evaluate the system's performance. Evaluation metrics, such as accuracy, sensitivity, specificity, precision, F1 score, Balance Classification Rate (BCR), Mathews Correlation Coefficient (MCC) and Kappa Coefficient (Kp), are calculated. Maximum accuracy, sensitivity, specificity, precision, F1 score, BCR, MCC, and Kp of 99.11 %, 98.99 %, 99.22 %, 99.09 %, 99.09 %, 99.10 %, 98.21 %, 98.21 %, respectively, were obtained for detection using fusion of shallow and deep features of Resnet18 pretrained network classified by BA optimized SVM classifier. Feature fusion performs better for classification task with accuracy, sensitivity, specificity, precision, F1 score, BCR, MCC and Kp of 97.31 %, 97.30 %, 98.65 %, 97.37 %, 97.34 %, 97.97%, 95.99 %, 93.95 %, respectively. The proposed brain tumour detection and classification framework using deep feature extraction from Resnet 18 pretrained network in conjunction with feature fusion and optimised ML classifiers can improve the system performance. Henceforth, the proposed work can be used as an assistive tool to aid the radiologist in automated brain tumour analysis and treatment.

Background and ObjectiveRecently, there has been a growing interest in the use of deep learning methods within the multi-modal domain of breast cancer research. Integrating multi-modal data for breast cancer prediction can generate richer and more diverse set of information, leading to a greater robustness in prediction outcomes as compared to single-modal approaches. This review comprehensively summarizes the advancements in multi-modal breast cancer research over the past 5 years and critically assesses the related opportunities and challenges, serving as a valuable reference for future studies. The application of deep learning techniques to the processing of multi-modal breast cancer data is discussed in depth, and the latest strategies and potential future directions in this area are examined.MethodsA systematic analysis of studies on deep learning methods for breast cancer diagnosis based on multi-modal data was conducted. A comprehensive literature search was performed across PubMed, Web of Science, Cochrane Library, and Google Scholar for studies published between January 2019 and April 2025. To ensure the representativeness of the included research, studies were evaluated according to three aspects: types of multi-modal data used, the fusion strategies adopted, and their clinical relevance.Key Content and FindingsThis review systematically traces the development of deep learning approaches for multi-modal breast cancer data, from foundational to more advanced methodologies. First, the paper categorizes common data types and core tasks related to breast cancer prediction. Subsequently, it classifies multi-modal data fusion strategies into three types—feature-level fusion, decision-level fusion, and hybrid fusion—providing a detailed explanation of the prediction steps for each category and comparing their effectiveness. Finally, the common challenges in multi-modal breast cancer research and insights into potential directions for future research are identified and discussed.ConclusionsAt present, although numerous deep learning–based multi-modal studies on breast cancer have been proposed, multi-modal fusion remains in the exploratory stage. Future research should focus on addressing the scarcity of high-quality public datasets, as well as developing more robust network architectures and adaptive fusion strategies to better capture complementary information across modalities.

A brain Magnetic resonance imaging (MRI) scan of a single individual consists of several slices across the 3D anatomical view. Therefore, manual segmentation of brain tumors from magnetic resonance (MR) images is a challenging and time-consuming task. In addition, an automated brain tumor classification from an MRI scan is non-invasive so that it avoids biopsy and make the diagnosis process safer. Since the beginning of this millennia and late nineties, the effort of the research community to come-up with automatic brain tumor segmentation and classification method has been tremendous. As a result, there are ample literature on the area focusing on segmentation using region growing, traditional machine learning and deep learning methods. Similarly, a number of tasks have been performed in the area of brain tumor classification into their respective histological type, and an impressive performance results have been obtained. Considering state of-the-art methods and their performance, the purpose of this paper is to provide a comprehensive survey of three, recently proposed, major brain tumor segmentation and classification model techniques, namely, region growing, shallow machine learning and deep learning. The established works included in this survey also covers technical aspects such as the strengths and weaknesses of different approaches, pre- and post-processing techniques, feature extraction, datasets, and models’ performance evaluation metrics.

CORR Synthesis: When Should the Orthopaedic Surgeon Use Artificial Intelligence, Machine Learning, and Deep Learning?

BackgroundAccurate classification of brain tumors is a major challenge in neuro‐oncology, as the heterogeneity of tumor morphology and the overlap of radiological features limit the effectiveness of conventional diagnostic approaches. Early and reliable tumor characterization is essential for treatment planning, prognosis, and improved patient outcomes. Recent advances in artificial intelligence (AI) have enabled the development of deep learning frameworks that can augment radiological interpretation and support clinical decision‐making.ObjectiveThis study proposes and validates a hybrid computational framework that integrates convolutional neural networks (CNNs) with graph convolutional networks (GCNs) for automated classification of brain tumors from magnetic resonance imaging (MRI).MethodsA publicly available Kaggle‐based MRI dataset was utilized, consisting of four categories: glioma, meningioma, pituitary tumor, and no‐tumor. The proposed pipeline incorporated systematic preprocessing, transfer learning via the InceptionV3 architecture for hierarchical feature extraction, graph construction to model inter‐feature relationships, and GCN‐based relational learning for final classification. Hyperparameter optimization was performed using Particle Swarm Optimization (PSO) to improve generalizability.ResultsThe experimental evaluation achieved an overall classification accuracy of 92.91%. Class‐specific performance analysis demonstrated particularly high diagnostic accuracy in the no‐tumor group (F1‐score: 0.9963) and pituitary tumor group (F1‐score: 0.9599). The incorporation of PSO tuning further improved the validation accuracy to 94.23%. The hybrid CNN–GCN framework exhibited robustness against imaging artifacts and irregular tumor boundaries, conditions that commonly challenge conventional classification techniques.ConclusionThe integration of CNN‐based hierarchical feature extraction with GCN‐based relational reasoning provides a significant advancement in automated brain tumor classification. This reproducible and intelligent diagnostic pipeline demonstrates strong potential for clinical translation by enhancing diagnostic precision, reducing radiologist workload, and facilitating timely therapeutic interventions. The findings support the integration of graph‐based deep learning systems into smart healthcare ecosystems, where AI‐assisted diagnostic tools can contribute to improved outcomes in neuro‐oncology.

Brain tumor detection is a significant problem in medical diagnostics since early and accurate detection improves patient outcomes. Traditional tumor identification techniques often depend on manual interpretation of medical examination, which can be time-consuming and prone to humanerror. Algorithms based on deep learning have emerged in recent years as a viable way to automateand enhance brain tumor identification using medical imaging data. This paper conveys an extensive look on the use of deep learning for brain tumor identification. A Convolutional NeuralNetwork(CNN) architecture is put forward to reach minimum accuracy of 97% and maximum of 100%, using its abilities to automatically learn hierarchical attributes from medical imagery that involve Magnetic Resonance Imaging(MRI) scans. To learn discriminative features suggestive oftumor presence, the suggested CNN framework is trained an extensive collection of labeled brainMRI images. The findings from experiments show that the proposed deep learning approach works. The trained CNN is quite good at differentiating between tumor and non-tumor regions in brain scans. Furthermore, cross-validation and unbiased evaluation are used to assess the model’scapacity to generalize to data that was previously unavailable. Deep learning in brain tumor identification has the potential to greatly enhance diagnostic accuracy, reduce human error, and speed up decision-making. As deep learning research advances, future studies may look at the amalgamation of multi-modal imaging data, transfer learning, and ensemble techniques in order to boost the robustness and generalizability of brain tumor diagnosis. The proposed deep learning-based brain tumor detection system offers the potential for improving medical professionals’ capacity to properly and instantly diagnose brain tumors, ultimately leading to improvements in patient care and outcomes. Keywords: Brain Tumor detection, Diagnosis, Deep Learning, Convolutional NeuralNetworks, Pooling, MRI Dataset

Today's digital health aims to provide an improved efficiency of healthcare delivery, and personalized, and timely disease care. Cardiovascular Disease (CVD) is a leading cause of death worldwide. In the United States, 1 out of 3 adults have some form of CVD. It is projected that nearly half of the US population will have at least one type of CVD by 2035, with total direct and indirect costs potentially surpassing $1 trillion (1-3). Medical imaging data encompass multiple modalities that are primarily utilized in silos. These include Computed Tomography (CT), Magnetic Resonance Imaging (MRI), CT-derived fractional flow reserve (CT-FFR), cardiac MRI, whole-heart dynamic 3D cardiac MRI perfusion, 3D cardiac MRI late gadolinium enhancement, cardiac positron emission tomography (PET), echocardiography and coronary angiography. However, only a few modalities are utilized in hybrid configurations, such as Positron Emission Tomography combined with Computed Tomography (PET/CT),Single-Photon Emission Computed Tomography combined with CT (SPECT/CT), Echocardiography and invasive angiography. Integrating these different imaging modalities becomes a burden on clinicians as it can lead to added complexity, potential inaccuracies, and increased healthcare costs. This research topic focused on fusion techniques that enable the integration and modeling of these multiple modalities to offer complementary information that will help improve CVD care. These modalities will leverage Machine Learning (ML) and Deep Learning (DL) techniques as well as other state-of-the-art techniques. Following are some insights and findings from this research topic: Milosevic et al.) conducted a systematic and comprehensive review on the state-of-the-art multi-modal medical data fusion in the context of CVD (4). Their review indicated that there are limited open multimodal datasets that are constrained both in size and modality scope. This scarcity of open datasets of labeled pathologies contributes to the comparatively few published papers on the diagnosis or prediction of cardiovascular diseases and conditions. The review indicated that over the last 5 years, there has been a considerable amount of work in artificial intelligence employing fusion techniques of multi-modal imaging involving various magnetic resonance imaging (MRI) and CT scans. However, the integration of modalities like x-ray, echocardiography, and non-imaging modalities remains relatively scarce.