Abstract Polycrystalline diamond compact (PDC) bit accounts for the most drilling footage in the development of deep and geothermal resources. The goal of this paper is to investigate the PDC cutter-rock interaction and reveal the rock fragmentation mechanism. A series of loading and unloading tests are conducted to obtain the curves of contact force versus penetration displacement. A single practical PDC cutter is fixed on the designed clamping devices that are mounted on the servo experiment system TAW-1000 in the tests. The craters morphology and quantified data were obtained by scanning the fragmented rock specimen using a three-dimensional morphology scanner. Finally, a numerical model is established to get the stress and deformation fields of the rock under a single PDC cutter. The results show that there are two kinds of failure modes, i.e., brittle failure and plastic failure, in the loading process. Marble is more prone to brittle fracture and has the lowest specific energy, followed by shale and granite. The brittle failure in marble mainly occurs behind the cutter while that happens ahead of the cutter for shale. Curves of contact force versus penetration displacement illustrate that a cutter with a back rake angle of 40 deg has a better penetration result than that with a back rake angle of 30 deg. Enhancing loading speed has a positive effect on brittle fragmentation. The distribution of von Mises stress indicates the initiation point and direction, which has a good agreement with the experiment. The research is of great significance for optimizing the PDC bit design and increasing the rate of penetration.