Solar photovoltaic (SPV) arrays are crucial components of clean and sustainable energy infrastructure. However, SPV panels are susceptible to thermal degradation defects that can impact their performance, thereby necessitating timely and accurate fault detection to maintain optimal energy generation. The considered case study focuses on an intelligent fault detection and diagnosis (IFDD) system for the analysis of radiometric infrared thermography (IRT) of SPV arrays in a predictive maintenance setting, enabling remote inspection and diagnostic monitoring of the SPV power plant sites. The proposed IFDD system employs a custom-developed deep learning approach which relies on convolutional neural networks for effective multiclass classification of defect types. The diagnosis of SPV panels is a challenging task for issues such as IRT data scarcity, defect-patterns’ complexity, and low thermal image acquisition quality due to noise and calibration issues. Hence, this research carefully prepares a customized high-quality but severely imbalanced six-class thermographic radiometric dataset of SPV panels. With respect to previous approaches, numerical temperature values in floating-point are used to train and validate the predictive models. The trained models display high accuracy for efficient thermal anomaly diagnosis. Finally, to create a trust in the IFDD system, the process underlying the classification model is investigated with perceptive explainability, for portraying the most discriminant image features, and mathematical-structure-based interpretability, to achieve multiclass feature clustering.