A weak negative thermal quenching (NTQ) effect of nitrogen vacancy centers in nitrogen-doped diamond is investigated by low-temperature photoluminescence technology. Results show that a weak NTQ phenomenon of the NV− centers was observed in the range of 10–40 K. Aiming to explain the observed NTQ effect, three hypotheses are proposed to classify the underlying physical mechanism, including (a) electron transfer from nitrogen atom to NV centers, (b) electron transfer between NV0 and NV− centers, and (c) the shallow traps of charge carriers in diamond. What is more, with incorporating sulfur, sulfur-nitrogen complex defects are formed with an impurity level of 0.65 eV below the conduction band. The activation energies of nonradiative recombination of NV− centers were obviously decreased with doping sulfur, which maybe related to the substitutional nitrogen (deep donor) replaced by sulfur-nitrogen complex structure defect (relatively shallow donor). The presented work provides a comprehensive understanding on the physical mechanism of NTQ effect of nitrogen vacancy centers in diamond, which would provide guidance for application of NV centers, especially in low temperatures.