Nanocrystalline diamond films with different grain sizes were synthesized on Si substrate by the hot cathode direct current plasma chemical vapor deposition method with different compositions of CH4/Ar/H2 gas mixture. The morphology and microstructure of the obtained products were characterized by scanning electron microscopy, atomic force microscopy, high-resolution transmission electron microscopy, X-ray diffraction, and Raman spectroscopy. The results showed that the composition of the CH4/Ar/H2 gas mixture affected greatly the grain sizes, surface smoothness, and crystal quality of the nanocrystalline diamond films. With the increase of CH4 concentration, the grain sizes got smaller and the surface of the films got smoother. However, the grain sizes got smaller and the crystal quality was weakened with the increase of Ar concentration. In the process of deposition, SiC was formed at the substrate before the growth of nanocrystalline diamond films, which increased the secondary nucleation of nanocrystalline diamonds. To obtain the nanocrystalline diamond films with uniform small grain sizes, good surface smoothness, and little non-diamond phase, the optimal composition of the CH4/Ar/H2 gas mixture is 6/70/30 and 7/70/30.