Integration of ultrasound and mammogram for multimodal classification of breast cancer using hybrid residual neural network and machine learning

Abstract

Breast cancer (BC) is one of the topmost causes of mortality in women all over the world. Early detection and classification of the tumor allow proper treatment of patients and chances of survival. In this article, we propose a hybrid residual neural network (ResNet) and machine learning framework and integrate the features of both mammography (MG) and ultrasound (US) images to perform the multimodal classification of BC images as benign or malignant. The features are extracted automatically from the input images of each modality using the residual neural network from the average pooling layer. Next, the feature level fusion is carried out to obtain a feature vector by combining features of MG & US. Finally, the multimodal classification is performed using the support vector machine (SVM) as a classifier. Experiments are performed on a real-time dataset collected from patients who have undergone both MG and US examinations. The classification accuracy obtained for the multimodal approach with SVM is 99.22%, which is higher than unimodal systems. Results show that the proposed multimodal approach performs better in classifying breast tumors than unimodal mammogram and ultrasound systems.