The KYb(MoO4)2: Er3+, Y3+ phosphors were synthesized by high temperature solid state reaction method. By further doping Y3+ ions, the near-infrared (NIR, 804 nm, 4I9/2 → 4I15/2) luminescence of KYb(MoO4)2: 0.1 % Er3+ phosphor is greatly enhanced. Based on the upconversion luminescence (UCL) spectra, the optimal doping concentration of Y3+ ions were 7.5 %. Its near-infrared UCL intensity is about 26.9 times that of undoped Y3+ ions. Through a series of characterization methods, we find that the enhancement of NIR upconversion emission is due to the generation of defect bands. It plays a crucial role in facilitating the transfer of energy from green light level (2H11/2, 4S3/2) to red light level (4F9/2) and near-infrared level. To explain the luminescence mechanism, the power dependence, UV-VI-NIR absorption spectra, excitation spectra at 804 nm and emission spectra at 380 nm were studied. In addition, the 804 nm single near-infrared UCL intensity of KYb (MoO4)2: Er3+, Y3+ shows a linear variation with temperature in the temperature range of 298 K–673 K. The maximum sensitivity is 0.13 % K−1. This study provides a new method and theoretical support for the application of near-infrared UCL in optical temperature measurement.