Breast cancer is a condition where the irregular growth of breast cells occurs uncontrollably, leading to the formation of tumors. It poses a significant threat to women’s lives globally, emphasizing the need for enhanced methods of detecting and categorizing the disease. In this work, we propose an Ensemble Deep Convolutional Neural Network (EDCNN) model that exhibits superior accuracy compared to several transfer learning models and the Vision Transformer model. Our EDCNN model integrates the strengths of the MobileNet and Xception models to improve its performance in breast cancer detection and classification. We employ various preprocessing techniques, including image resizing, data normalization, and data augmentation, to prepare the data for analysis. By following these measures, the formatting is optimized, and the model’s capacity to make generalizations is improved. We trained and evaluated our proposed EDCNN model using ultrasound images, a widely available modality for breast cancer imaging. The outcomes of our experiments illustrate that the EDCNN model attains an exceptional accuracy of 87.82% on Dataset 1 and 85.69% on Dataset 2, surpassing the performance of several well-known transfer learning models and the Vision Transformer model. Furthermore, an AUC value of 0.91 on Dataset 1 highlights the robustness and effectiveness of our proposed model. Moreover, we highlight the incorporation of the Grad-CAM Explainable Artificial Intelligence (XAI) technique to improve the interpretability and transparency of our proposed model. Additionally, we performed image segmentation using the U-Net segmentation technique on the input ultrasound images. This segmentation process allowed for the identification and isolation of specific regions of interest, facilitating a more comprehensive analysis of breast cancer characteristics. In conclusion, the study presents a creative approach to detecting and categorizing breast cancer, demonstrating the superior performance of the EDCNN model compared to well-established transfer learning models. Through advanced deep learning techniques and image segmentation, this study contributes to improving diagnosis and treatment outcomes in breast cancer.