This paper aims to reveal the relationship between the high-speed gear wear and nonlinear dynamics for a wind turbine gearbox considering the tooth contact temperature. The temperature and deformation of the tooth surface are calculated respectively using the Blok's flash temperature theory and thermal deformation formula, and the meshing stiffness caused by the tooth contact temperature is obtained using the Hertz theory. The wear depth and time-varying meshing stiffness of high-speed gears are calculated using the Archard's equation and potential energy method, respectively. The nonlinear dynamic model of the gearbox is established to study the characteristics, and the frequency domain values of gear wear are compared with experimental data. The results show that the tooth contact temperature makes the time-varying meshing stiffness decrease, then affecting the dynamics of the system, the stability decreases with the increase of comprehensive transmission error, and for the gearbox considering the contact temperature, the gear wear makes the chaotic motion occur in advance and its region increase obviously. The research provides a basis for the wear fault diagnosis and detection of gear transmission systems.