We study coherent diffractive photon and vector meson production in electron-proton and electron-nucleus collisions within the Color Glass Condensate effective field theory. We show that electron-photon and electron-vector meson azimuthal angle correlations are sensitive to non-trivial spatial correlations in the gluon distribution of the target, and perform explicit calculations using spatially dependent McLerran-Venugopalan initial color charge configurations coupled to the numerical solution of small $x$ JIMWLK evolution equations. We compute the cross-section differentially in $Q^2$ and $|t|$ and find sizeable anisotropies in the electron-photon and electron-$\mathrm{J}/\psi$ azimuthal correlations ($v_{1,2} \approx 2 - 10 \%$) in electron-proton collisions for the kinematics of the future Electron-Ion Collider. In electron-gold collisions these modulations are found to be significantly smaller ($v_{1,2} <0.1 \%$). We also compute incoherent diffractive production where we find that the azimuthal correlations are sensitive to fluctuations of the gluon distribution in the target.