The precise location and size of distributed photovoltaics (PVs) is critical to infer the actual installed capacity and assess the remaining PV generation potential, and is therefore the cornerstone of strategic planning for distributed PV deployment. However, identifying small-scale distributed PVs in complex contexts from high spatial resolution remote sensing (HSRRS) images to obtain their information remains an issue. In this study, we propose an advanced deep learning model, called PV Identifier, to enhance the identification accuracy of small-scale PV systems from HSRRS images. PV Identifier uses a fine-grained feature layer (FFL) compatible with the size of PVs to improve the detection capability of the small-scale distributed PVs. At the same time, it effectively distinguishes between PVs and similar background using a novel semantic constraint module (SCM). We test PV Identifier on a distributed PV dataset in California. Experiments show that the inclusion of the FFL positively affects the model's sensitivity to small distributed PVs. Specifically, the PV Identifier with the FFL increases the Recall of identifying residential rooftop PVs by 1.9% compared to the model without the FFL. In addition, the integration of the SCM effectively improves the model's ability to locate residential rooftop PVs in complex environments, resulting in a 1.8% increase in the corresponding Precision. Compared to the four commonly used segmentation models, PV Identifier exhibits superior identification performance for residential rooftop PVs and commercial and industrial PVs, with an Intersection over Union (IoU) of 74.1% and 89.3%, respectively, which is at least 4.1% and 1.8% higher than other models. Overall, PV Identifier provides a viable solution to the problem of identifying small-scale distributed PV in complex backgrounds from HSRRS images.