Reconstructing the 3D shape of objects using limited information from a single image is a complex and necessary task. The existing methods for reconstructing point clouds from a single image haven't effectively addressed the issue of background interference in the input image, thus greatly limited their practical application. To address the underlying issue, this paper proposes a 3D-ISRNet aiming to enhance the accuracy of point cloud reconstruction through image similarity retrieval. The 3D-ISRNet retrieves point clouds similar to the object and employs spatial and channel attention mechanisms for image encoding. This aims to enhance the features of the reconstruction area and improve reconstruction accuracy. After the fusion of retrieved point clouds with image features, the combined data undergoes a symmetric encoder to generate the reconstructed point clouds, thereby alleviating the constraints imposed by the initial point clouds on the reconstruction results. 3D-ISRNet demonstrates favorable results on public and real-world datasets.