Co2FeAl films with different thickness were prepared at different temperature by molecular beam epitaxy. Their dynamic magnetic property was studied by the time-resolved magneto-optical Kerr effect measurements. It is observed that the intrinsic damping factor of Co2FeAl for [100] orientation is not related to the film's thickness and magnetic anisotropy as well as temperature at high-field regime, but increases with structural disorder of Co2FeAl. The dominant contribution from the inhomogeneous magnetic anisotropy is revealed to be responsible for the observed extremely nonlinear and drastic field-dependent damping factors at low-field regime.