Thoracic diseases are a major source of mortality, often requiring diagnosis from plain chest X-rays. However, differentiating between complex conditions based on subtle radiographic patterns poses challenges even for experts. Recently, deep learning methods have shown promise in automating thoracic disease detection from chest radiographs. Many existing approaches focus on the diseased organs in the radiographs by utilizing spatial regions that contribute significantly to the model’s prediction. Expert radiologists, on the other hand, first identify the prominent region before determining whether those regions are abnormal or not. Therefore, incorporating localization information through deep learning models could result in significant improvements in automatic disease classification. Motivated by this, we have proposed a generalized weakly supervised Confidence-Aware Probabilistic Class Activation Map (CAPCAM) classification model that localizes anomalies for thoracic disease. The CAPCAM used CX-Ultranet as the backbone with the combination of Confidence Aware Network (CAN) and Anomaly Detection Network (ADN) without having any localization labeling. This learning from the backbone helps the model to utilize all components of the feature extracted and, therefore eliminating the need to train them individually reducing the time taken. We have experimentally shown that the proposed CAPCAM method sets a new state-of-the-art benchmark by achieving accuracy in terms of Intersection of bounding box (IoBB) in the range of 85 % - 94 %, and Dice scores in the range of 88 %-90 % for all thirteen diseases on two publicly available large-scale CXR datasets–NIH, Stanford and CheXpert. Testing across different noise levels and different levels of blurred level assessed real-world viability. We have also added a layer of explainability to show how the image is processed. This study demonstrates deep learning’s potential to augment radiologists’ decision-making by providing fast, accurate automated aids for thoracic disease diagnosis. The proposed CAPCAM model could be readily translatable to improve clinical workflows.