The stress release method with rock slotting in the bottom hole by ultra-high-pressure water jet is proposed as a solution to the problem of increased rock strength due to high in-situ stress. However, the failure mode of rock with annular grooves under the simultaneous action of geostatic stress and hydrostatic pressure is still unclear. The failure characteristics, penetration force, failure mode and deformation characteristics of rock under geostatic stress are studied by numerical simulation. The results showed that the presence of an annular groove changes the rock failure mode. The radial crack propagation direction dominated by extrusion shear is deflected and will be connected to the peripheral area of the groove’s bottom. It becomes the primary crack propagation dominated by tensile shear, controlled by the crack formation zone and the stress concentration zone at the peripheral area of the groove’s bottom. The annular groove can significantly reduce the peak penetration force by about 72%. There is a critical value for the influence of groove depth on the peak penetration force. The presence of an annular groove changes the load-penetration curves from ductile failure to brittle failure mode. The groove modifies the stress propagation mode, altering the rock response and crack propagation zones. In addition, the annular groove decreases the radial force that restricts the extension of rock in the surrounding area, allowing the rock to expand more easily in the surrounding area. Therefore, this stress release method and the study’s findings are instructive for improving rock-breaking efficiency and gaining a deeper understanding of the technique.