Viscoelastic (VE) dampers can effectively mitigate the vibration of structures under dynamic loading depending on better energy dissipating capability and are widely applied in practical engineering. On the basis of the studies on the mechanical properties of the VE material, the nonlinear behaviors of the VE damper are frequency dependent which can be elucidated by the changes in storage modulus and loss factor parameters. In this paper, experimental results of the VE damper reveal that the mechanical performance of the VE damper is correlative with the displacement amplitude. Therefore, a modified fractional derivative model is proposed to reflect the softening behavior of displacement amplitude on the VE damper. This result demonstrates that the modified model is reasonably practicable and can accurately evaluate the characteristic parameters of the VE damper. In order to apply the proposed constitutive model to finite element software, an equivalent method is presented to transform this model into the generalized Maxwell model. A three-dimensional finite element model of the VE damper is established with ABAQUS for numerical analysis. And compared with the experimental results, it is indicated that the equivalent generalized Maxwell model is available. This work verifies the conversion of the fractional derivative model to the linear VE model is feasible and provides a useful approach for the utilization of the fractional derivative model of the VE damper in finite element software.