Under the influence of continuous periodic changes of various meteorological factors, there will be an inhomogeneous temperature field inside the asphalt pavement, which will cause a continuous gradient viscoelasticity of asphalt concrete with depth. In order to perform the structural mechanics analysis accurately, the temperature field gradient effect and the temperature-induced gradient viscoelasticity distribution inside the asphalt pavement should be revealed first. In this research, based on the heat conduction theory, a temperature field finite element analysis (FEA) model of a semi-rigid base asphalt pavement was established to investigate the distribution behavior of the temperature gradient along the depth. Dynamic modulus tests were conducted to survey temperature effect on the viscoelastic parameters of asphalt concrete. Eventually, the intrinsic relationship between temperature, depth, and viscoelastic parameters was integrated, and the relaxation time shift method was proposed to quantitatively characterize the temperature-induced gradient viscoelasticity at different depths. It is found that the temperature varies in a cubic parabolic fashion along the depth under the most significant positive and negative temperature gradient conditions. The relaxation time shift method can be effectively used to characterize the distribution of the relaxation modulus and relaxation time spectrum of asphalt concrete under the influence of temperature. It is easy to be programmed to fully consider the inhomogeneous temperature field, and applicable to both traditional and gradient FEA. The change of parameters of generalized Maxwell model for asphalt concrete with temperature is essentially the result of relaxation spectrum shifting along the relaxation time axis at different temperatures. When the temperature is high, the viscosity decreases, the relaxation spectrum shifts to the direction of decreasing relaxation time, the relaxation modulus of asphalt concrete decreases, and vice versa.