Si ions 2.3 MeV are implanted into nanodiamonds (NDs) at doses of 1×1012 ~ 1×1015 ions/cm2. The ion implantation not only creates silicon-vacancy (SiV) color centers but also reduces the size of the NDs from 50 nm to ~ 10 nm. As the Si dose is increased up to 1 × 1013 ions/cm2, the luminescence from the nitrogen-vacancy (NV) color centers in the ND initially increases. At higher dose rates, the luminescence from the NV color centers decreases. Due to the differences in the minimum ND size required for stable luminescence, the zero phonon line (ZPL) of the SiV color center appears after the luminescence from the NV center decreases dramatically. The ZPLs from both centers become almost negligible after Si ions have been implanted at doses higher than 5 × 1014 ions/cm2. These observations are explained by the reduced size of the NDs and the number of implanted Si ions, which is estimated based on SRIM simulations.