The rolling cutter with spherical inserted-tooth has been an effective tool for mechanical excavation. Significant research has been conducted on the optimization of the tooth shape parameters, tooth spacing, and row spacing of the alloy teeth on the cutter to improve the efficiency of mechanical tunneling and reduce the construction cost. In this study, three spherical teeth with different diameters were designed to perform indentation tests on Beishan granite under various penetration and spacing conditions. The rock fragmentation mechanisms of spherical teeth was analyzed through the fluorescence method, to optimally design the spherical inserted-tooth roller cutter. For Beishan granite, when the tooth spacing and row spacing were about 30 mm and 35 mm respectively, the spherical tooth with a radius of 16 mm gained the highest rock breaking efficiency. Research results provide a guidance for the design of the inserted-tooth roller cutter and the force estimation of cutter head.