Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method

Abstract

Although the application of deep learning has greatly improved the performance of benign and malignant breast cancer classification algorithm, the accuracy of classification using only the pathological image has been unable to meet the requirements of clinical practice. Inspired by the real scene when the pathologist read the pathological image for diagnosis, in this paper, we propose a new hybrid deep learning method for benign and malignant breast cancer classification. From the perspective of multimodal data fusion, our method combines pathological image and structured data in the clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification. Thus, the proposed method can be useful for breast cancer diagnosis in real clinical practice. Experimental results based on our datasets show that the proposed method significantly outperforms the state-of-the-art methods in terms of overall classification accuracy.