We study a single field axion inflation model in the presence of an SU(2) gauge field with a small vev. In order to make the analysis as model-independent as possible, we consider an arbitrary potential for the axion that is able to support the slow-roll inflation. The gauge field is coupled to the axion with a Chern-Simons interaction $$ \frac{\lambda }{f}{F}_{\mu \nu}^a{\tilde{F}}_a^{\mu \nu } $$ where $$ \frac{\lambda }{f}\sim \frac{\mathcal{O}(10)}{M_{\mathrm{pl}}} $$ . It has a negligible effect on the background evolution, $$ \frac{\rho \mathrm{Y}\mathrm{M}}{M_{\mathrm{pl}}^2{H}^2}\lesssim {\upepsilon}^2 $$ . However, its quantum fluctuations make a significant contribution to the cosmic perturbation. In particular, the gauge field has a spin-2 fluctuation which explicitly breaks the parity between the left- and right-handed polarization states. The chiral tensor modes are linearly coupled to the gravitational waves and lead to a circularly polarized tensor power spectrum comparable to the unpolarized vacuum power spectrum. Moreover, the scalar sector is modified by the linear scalar fluctuations of the gauge field. Since the spin-0 and spin-2 fluctuations of the SU(2) gauge field are independent, the gauge field can, at the same time, generate a detectable chiral gravitational wave signal and have a negligible contribution to the scalar fluctuations, in agreement with the current CMB observations.