Deep learning-based approaches have become widespread in medical fields and have achieved profound success in recent years. Nonetheless, most of these approaches cannot provide the certainty of their output numerically, and their output often has too many confidence levels. Therefore, they cannot be efficient and may even lead to irreparable damage. Accordingly, to tackle such a challenge, the Bayesian approximation and Ensemble learning techniques are deliberated as uncertainty quantification (UQ) methods. We apply and evaluate three UQ models regarding the classification of breast tumor tissue types. These methods are Mont Carlo-dropout (MCD), Bayesian Ensemble, and MCD Ensemble. Furthermore, to increase the classification's precision and eliminate the adverse effects of the smallness of the data collection in Wisconsin Diagnostic Breast Cancer (WDBC) used in the present research, transfer learning technique and pre-trained Convolutional Neural Network (DenseNet121) are considered. Three proposed models are compared based on their ability to quantify the reliability of classification according to novel performance metrics employed to assess estimated uncertainty. Our quantitative and qualitative examinations in the study demonstrate that models show high uncertainty in misclassifications, which is vital in determining the rate of medical diagnosis risks. As a result, by using these new assessment criteria, we intend to ascertain when can trust the output of the deep neural network. Moreover, in the examination, we witness more reliability in uncertainty quantification of the Bayesian Ensemble model.