Computer-generated holography (CGH) is an effective light field manipulation technique based on diffractive optics. Deep learning provides a promising way to break the trade-off between quality and speed in the phase-only hologram (POH) generation process. In this paper, a neural network called BERDNet is proposed for high-quality and high-speed POH generation. A high-quality POH dataset without speckle noise and shifting noise is generated by the band-limited bidirectional error diffusion (BERD) algorithm. Based on the dataset, BERDNet is trained to learn the potential hologram coding method for real-time POH prediction. Furthermore, the training process is constrained by both data loss and physical loss, so it is necessary to explore higher-fidelity reconstructions that are more consistent with the bandwidth limitation. Finally, the POHs of numerical reconstructions with an average of 23.13 dB PSNR can be obtained in 0.037 s, achieving 1-2 orders of magnitude acceleration. Experimental reconstructions validated the generalization of the BERDNet.