III-Nitride semiconductor-based quantum dots (QDs) play an essential role in solid-state quantum light sources because of their potential for room-temperature operation. However, undesired background emission from the surroundings deteriorates single-photon purity. Moreover, spectral diffusion causes inhomogeneous broadening and limits the applications of QDs in quantum photonic technologies. To overcome these obstacles, it is demonstrated that directly pumping carriers to the excited state of the QD reduces the number of carriers generated in the vicinities. The polarization-controlled quasi-resonant excitation is applied to InGaN QDs embedded in GaN nanowire. To analyze the different excitation mechanisms, polarization-resolved absorptions are investigated under the above-barrier bandgap, below-barrier bandgap, and quasi-resonant excitation conditions. By employing polarization-controlled quasi-resonant excitation, the linewidth is reduced from 353 to 272µeV, and the second-order correlation value is improved from 0.470 to 0.231. Therefore, a greater single-photon purity can be obtained at higher temperatures due to decreased linewidth and background emission.