We calculate the primordial correlation of gravitons with an abelian gauge field non-minimally coupled through a dynamical dilaton field or a volume moduli during inflation in the early universe. In particular, we compute the cross-correlation of a tensor mode with two gauge field modes and the corresponding correlation functions for the associated magnetic and electric fields using the in-in formalism. Moreover, using semi-classical methods, we show that the three-point cross-correlation functions satisfy new consistency relations (soft theorems) in the squeezed limit. Our findings exhibit a complete agreement of the full in-in results with the new consistency relations. An interesting consequence of our scenario is the possibility of a novel correlation of the primordial tensor mode with the primordial curvature perturbation induced by higher order quantum gravity corrections. The anisotropic background created by long wavelength gauge field modes makes this correlation function non-vanishing. Finally, we discuss how these three-point correlation functions are imprinted on cosmological observables today and the applications to scenarios of inflationary magnetogenesis.