The interference-less coded aperture correlation holography is a non-scanning, motionless, and incoherent technique for imaging three-dimensional objects without two-wave interference. Nevertheless, a challenge lies in that the coded phase mask encodes the system noise, while traditional reconstruction algorithms often introduce unwanted surplus background components during reconstruction. A deep learning-based method is proposed to mitigate system noise and background components simultaneously. Specifically, this method involves two sub-networks: a coded phase mask design sub-network and an image reconstruction sub-network. The former leverages the object's frequency distribution to generate an adaptive coded phase mask that encodes the object wave-front precisely without being affected by the superfluous system noise. The latter establishes a mapping between the autocorrelations of the hologram and the object, effectively suppresses the background components by embedding a prior physical knowledge and improves the neural network's adaptability and interpretability. Experimental results demonstrate the effectiveness of the proposed method in suppressing system noise and background components, thereby significantly improving the signal-to-noise ratio of the reconstructed images.