In axion gauge field inflation an axion-like particle driving cosmic inflation is coupled to the Chern-Simons density of an Abelian or non-Abelian gauge group. In the case of a non-Abelian gauge group, this can lead to the formation of a stable, homogeneous and isotropic gauge field background. We study the dynamics of the inflaton and gauge fields in terms of the two effective coupling parameters: the gauge coupling and the axion decay constant. Starting from the Bunch-Davies vacuum in the far past, we find that the non-trivial gauge field background arises only significantly after the cosmic microwave background (CMB) scales have left the horizon. At these scales, the model thus closely resembles Abelian axion inflation, thereby naturally reconciling the tension of non-Abelian axion gauge field inflation with the latest CMB observations. We further consider two exemplary UV-completions of this setup: multiple Peccei-Quinn axions and axion monodromy in string theory. In both cases we find that the majority of the parameter space is excluded by theoretical or observational constraints. The remaining parameter space can be divided into three regimes. (i) For small gauge couplings we recover natural inflation. For large gauge couplings the non-Abelian gauge theory either (ii) mimics the Abelian theory or (iii) non-linear interactions prohibit a linear analysis of the gauge field perturbations.