Color centers in diamonds have shown excellent potential for applications in quantum information processing, photonics, and biology. Here we report chemical vapor deposition (CVD) growth of nanocrystalline diamond (NCD) films as thin as 5-6 nm with photoluminescence (PL) from silicon-vacancy (SiV) centers at 739 nm. Instead of conventional 4-6 nm detonation nanodiamonds (DNDs), we prepared and employed hydrogenated 2 nm DNDs (zeta potential = +36 mV) to form extremely dense (∼1.3 × 1013 cm-2), thin (2 ± 1 nm), and smooth (RMS roughness < 0.8 nm) nucleation layers on an Si/SiOx substrate, which enabled the CVD growth of such ultrathin NCD films in two different and complementary microwave (MW) CVD systems: (i) focused MW plasma with an ellipsoidal cavity resonator and (ii) pulsed MW plasma with a linear antenna arrangement. Analytical ultracentrifuge, infrared and Raman spectroscopies, atomic force microscopy, and scanning electron microscopy are used for detailed characterization of the 2 nm H-DNDs and the nucleation layer as well as the ultrathin NCD films. We also demonstrate on/off switching of the SiV center PL in the NCD films thinner than 10 nm, which is achieved by changing their surface chemistry.