Digital holography is a method of recording light waves emitted from an object as holograms and then reconstructing the holograms using light wave propagation calculations to observe the object in three dimensions. However, a problem with digital holography is that unwanted images, such as conjugate images, are superimposed as the hologram is reconstructed to create an observed image. In particular, the superimposition of conjugate light on the observed image is caused by the imaging device’s ability to record just the intensity distribution of light rather than the phase distribution of light. In digital holography, it has been shown that unwanted light can be eliminated by the phase-shift method. However, it is difficult to apply the phase-shift method to digital holographic microscopy (DHM), which takes only one shot of light intensity. Alternatively, machine learning methods called deep learning have been actively studied in recent years for image-related problems, with image transformation as an example. Furthermore, a method that combines digital holography and deep learning has been proposed to perform image transformation to remove conjugate images using deep learning on the reconstructed image of a hologram. In this study, we generated a pair of holograms with only light intensity distribution and holograms with complex amplitude by simulating light wave propagation, trained U-Net to perform image transformation that adds phase information to the hologram with only light intensity distribution, and proposed a method for phase retrieval and conjugate image removal for holograms using the learned U-Net. To verify the effectiveness of the proposed method, we evaluated the image quality of the reconstructed image of holograms before and after processing by U-Net. Results showed that the peak signal-to-noise ratio (PSNR) increased by 8.37 [dB] in amplitude and 9.06 [dB] in phase. The amplitude and phase of the structural similarity index (SSIM) increased by 0.0566 and 0.0143, respectively. Furthermore, the results of applying the proposed method to holograms captured by actual digital holography optics showed the effectiveness of the proposed method in eliminating conjugate images in the reconstructed images. These results show that the proposed method is capable of phase retrieval of holograms in a single shot without the need for a complex optical system. This is expected to contribute to the field of portable DHMs and other applications that require compact and simple optical systems.