TimeDistributed-CNN-LSTM: A Hybrid Approach Combining CNN and LSTM to Classify Brain Tumor on 3D MRI Scans Performing Ablation Study

Abstract

Identification of brain tumors and accurate grading at an early stage are crucial in cancer diagnosis, as a timely diagnosis can increase the chances of survival. Considering the challenges and risks of tumor biopsies, noninvasive imaging procedures such as Magnetic Resonance Imaging (MRI) are extensively used in analyzing brain tumors. Recent advances in the field of medical imaging with deep learning using three dimensional (3D) MRI is aiding the clinical experts significantly in the diagnosis of brain tumor. In this study, three BraTS MRI datasets named BraTS 2018, BraTS 2019 and BraTS 2020 are employed to classify brain tumor into high-grade glioma (HGG) and low-grade glioma (LGG) where each of the datasets contains four different sequences of 3D MRI brain images named T1-weighted MRI (T1), T1-weighted MRI with contrast enhancement (T1ce), T2-weighted MRI (T2), and Fluid Attenuated Inversion Recovery (FLAIR) for a single patient. This research is composed of two approaches where, in the first part, we propose a hybrid deep learning model named TimeDistributed-CNN-LSTM (TD-CNN-LSTM) combining 3D Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) where each layer of the architecture is wrapped with a TimeDistributed function. The objective of developing the hybrid model is to consider all the four 3D MRI sequences of each patient as a single input data because every sequence contains necessary information on tumor that can have an impact on improving cancer detection performance. However, interpreting all the MRI sequences together with optimal performance especially in 3D is quite challenging. Therefore, the model is developed with optimal configuration based on highest accuracy performing ablation study for layer architecture and hyper-parameters. In the second part, a 3D CNN model is trained respectively with each of the MRI sequences to compare the performance with TD-CNN-LSTM model. In this regard, for both of the models, BraTS 2018 and BraTS 2019 are combined to increase the number of images and used as train dataset where BraTS 2020 dataset is employed as the test dataset. Moreover, before training the models the datasets is preprocessed to ensure the highest performance. Our results of these two approaches demonstrate that the TD-CNN-LSTM network outperforms 3D CNN achieving the highest test accuracy of 98.90%. Later, to evaluate the performance consistency, the TD-CNN-LSTM model is evaluated with K-fold cross validation using different k values. The approach of putting together all the MRI sequences at a time using hybrid CNN-LSTM approach with good generalization capability to classify brain tumor can be used effectively in future Computer Aided Diagnosis (CAD) based research which can aid radiologists in medical diagnostics.