Breast cancer is a significant health concern due to its aggressive nature and high mortality rates. Early detection is crucial to improving patient outcomes. Thermography, a non-invasive and cost-effective method, utilizes heat from the breast surface to detect abnormalities, with the Database for Mastology Research (DMR) leveraging infrared images for research purposes. Deep learning (DL), particularly convolutional neural networks (CNNs), shows promise in accurately classifying breast cancer images. However, CNNs face challenges with hyperparameters. To address these challenges, this article proposes a new DL model, the optimized DenseNet model (LFR-COA-DenseNet121-BC), incorporating a boosted metaheuristic algorithm called LFR-COA. This algorithm, a developed version of the Coati Optimization Algorithm (COA), integrates Random opposition-based learning (ROB), Brownian motion, and Lévy Flight (LF) schemes. The proposed model achieved impressive results, accurately classifying 99.97% of the test set. Comparison with established models such as VGG16, VGG19, DenseNet201, InceptionV3, Xception, and MobileNet revealed superior performance of the LFR-COA-DenseNet121-BC model, with 99.97% accuracy, 99.96% sensitivity, and 99.9% specificity. Further comparison with COA-DenseNet121 highlighted the superiority of LFR-COA-DenseNet-BC. In addition, LFR-COA was evaluated at the IEEE Congress on Evolutionary Computation held in 2022 (CEC 2022), and real-world medical scenarios showcased the effectiveness of LFR-COA compared to existing optimization algorithms. LFR-COA consistently outperformed the original COA algorithm and other well-known counterparts in various statistical, convergence, and diversity measures, affirming its efficacy in breast cancer classification.