Context. The Polarimetric and Helioseismic Imager on board the Solar Orbiter mission (SO/PHI) offers refocusing capabilities to cope with the strongly varying thermal environment of the optical system along the spacecraft’s elliptical orbit. The series of images recorded during in-flight focus calibrations can be employed for phase diversity analyses. Aims. In this work we infer the wavefront degradation caused by the thermo-optical effects in the High Resolution Telescope (HRT) from images taken during the fine and coarse focus scans performed in the commissioning phase of the instrument. The difference between these two series of images are mainly related to the employed defocused step (smaller for the fine scans) and the signal-to-noise ratio (higher for the coarse scans). We use the retrieved wavefronts to reconstruct the original scene observed during the calibration of the instrument. Methods. We applied a generalized phase diversity algorithm that allowed us to use several images taken with different amounts of defocus to sense the wavefront degradation caused by the instrument. The algorithm also uses information from both the inferred wavefront and the series of images to restore the solar scene. Results. We find that most of the retrieved Zernike coefficients tend to converge to the same value when increasing the number of images employed for PD for both the fine and the coarse focusing scans. The restored scenes also show signs of convergence, and the merit function is minimized more as K increases. Apart from a defocus, the inferred wavefronts are consistent for the two datasets (λ/10 − λ/11). For the fine scan images, the quiet-sun contrast improves from 4.5% for the original focused image up to about 10%. For the coarse scan images, the contrast of the restored scene is as high as 11%.