The use of thermal infrared imaging and the advancement of image processing techniques have developed fault diagnosis of Photo Voltaic (PV) panels without the need to incorporate expensive electrical detection circuitry. PV thermal image processing, supports in continuous monitoring of panels without disturbing the continuity of operation and mitigates the problem of manual inspection involved over a large area. The thermal pictures of photo voltaic panels are used to identify the nature of faults such as bypass diode, cell under fault condition, circuit which is under open condition, hot spot, polarization, Cracking, Corrosion, discoloring and delamination. An automatic PV Computer Aided Diagnosis (CAD) based condition monitoring systems with thermal image analysis is developed to identify and classify the different fault conditions such as Hotspot, Ethylene Vinyl Acetate (EVA) Discoloration, Delamination and Open Circuit. The proposed system involves a hybrid thermal pixel counting algorithm to identify the thermal patterns. The Extreme Gradient Boosting algorithm (XGBoost) is used to detect the numerical characteristics of the captured image thermal variations to support in classification. The experimental results show that the proposed Hybrid Thermal Pixel Counting (HTPC) Algorithm with XGBOOST Classifier gives better detection accuracy with improved computational speed.