Nanodiamonds (NDs) hosting nitrogen-vacancy (NV) centers are promising for applications of quantum sensing. Long spin relaxation times (T1 and T2) are critical for high sensitivity in quantum applications. It has been shown that fluctuations of magnetic fields due to surface spins strongly influence T1 and T2 in NDs. However, their relaxation mechanisms have yet to be fully understood. In this paper, we investigate the relation between surface spins and T1 and T2 of single-substitutional nitrogen impurity (P1) centers in NDs. The P1 centers located typically in the vicinity of NV centers are a great model system to study the spin relaxation processes of the NV centers. By employing high-frequency electron paramagnetic resonance spectroscopy, we verify that air annealing removes surface spins efficiently and significantly reduces their contribution to T1.