In this Letter, we investigate the effects of a time-dependent, short-ranged interaction on the long-time expansion dynamics of Fermi gases. We show that the effects of the interaction on the dynamics is dictated by how it changes under a conformal transformation, and derive an explicit criterion for the relevancy of time-dependent interactions near both the strongly and noninteracting scale invariant limits. In addition, we show that it is possible to engineer interactions that give rise to nonexponential thermalization dynamics in trapped Fermi gases. To supplement the symmetry analysis, we perform hydrodynamic simulations to show that the moment of inertia of the trapped gas indeed follows a universal time dependence that is determined jointly by the conformal symmetry and time-dependent scattering length a(t). Our results should also be relevant to the dynamics of other systems that are nearly scale invariant and that are governed by a nonrelativistic conformal symmetry.