For light-element materials, X-ray phase contrast imaging provides better contrast compared to absorption imaging. While the Fourier transform method has a shorter imaging time, it typically results in lower image quality; in contrast, the phase-shifting method offers higher image quality but is more time-consuming and involves a higher radiation dose. To rapidly reconstruct low-dose X-ray phase contrast images, this study developed a model based on Generative Adversarial Networks (GAN), incorporating custom layers and self-attention mechanisms to recover high-quality phase contrast images. We generated a simulated dataset using Kaggle’s X-ray data to train the GAN, and in simulated experiments, we achieved significant improvements in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). To further validate our method, we applied it to fringe images acquired from three phase contrast systems: a single-grating phase contrast system, a Talbot-Lau system, and a cascaded grating system. The current results demonstrate that our method successfully restored high-quality phase contrast images from fringe images collected in experimental settings, though it should be noted that these results were achieved using relatively simple sample configurations.