The single polycrystalline diamond compact (PDC) cutter test is widely used to investigate the mechanism of rock-breaking. The generated cuttings and cutting force are important indexes reflecting the rock failure process. However, they were treated as two separate parameters in previous publications. In this study, through a series of rock block cutting tests, the relationship between them was investigated to obtain an in-depth understanding of the formation of cuttings. In addition, to validate the standpoints obtained in the aforementioned experiments, rock sheet cutting tests were conducted and the rock failure process was monitored by a high-speed camera frame by frame. The cutting force was recorded with the same sampling rate as the camera. By this design, every sampled point of cutting force can match a picture captured by the camera, which reflects the interaction between the rock and the cutter. The results indicate that the increase in cutting depth results in a transition of rock failure modes. At shallow cutting depth, ductile failure dominates and all the cuttings are produced by the compression of the cutter. The corresponding cutting force fluctuates slightly. However, beyond the critical depth, brittle failure occurs and chunk-like cuttings appear, which leads to a sharp decrease in cutting force. After that, the generation of new surface results in a significant decrease in actual cutting depth, a parameter proposed to reflect the interaction between the rock and the cutter. Consequently, ductile failure dominates again and a slight fluctuation of cutting force can be detected. As the cutter moves to the rock, the actual cutting depth gradually increases, which results in the subsequent generation of chunk-like cuttings. It is accompanied by an obvious cutting force drop. That is, ductile failure and brittle failure, one following another, present at large cutting depth. The transition of rock failure mode can be correlated with the variation of cutting force. Based on the results of this paper, the real-time monitoring of torque may be helpful to determine the efficiency of PDC bits in the downhole.