This paper proposes a new viscoelastic (VE) material damping device with hybrid nonlinear properties. Compared with traditional linear material dampers, the new nonlinear VE material damping device is characterized by its better energy dissipation and deformation capability. The series performance tests of the VE device are conducted, based on which the sources and variation law of material nonlinearities are analyzed. Five aspects of material nonlinearities are summarized, including the shape change of hysteresis loop caused by phase difference, the initial stiffness caused by large loading rate, the softening effect caused by high temperature and fatigue, and the softening and stiffening effect under large strain deformation. Then, a mechanic model for this device is proposed which considers multiple nonlinear effects of the material. For the verifications of the proposed mechanic model, a shaking table test on a steel frame equipped with the new VE devices is designed and performed. Based on the proposed mathematic mechanic model, the numerical mechanic model is implemented in the OpenSees software. The accuracy of the mechanic model is firstly verified by comparing the performance tests data with the simulation results. Then numerical model is also used to calculate the time history response of the shake table tested steel frame under earthquake loading. It is concluded that the mechanic model can well depict several nonlinear material behaviors of the new VE device, and the corresponding numerical model created in the open source calculation platform is reliable to be used to calculate nonlinear time history response of a structure equipped with the new VE material damping devices.