Cavitation erosion on sedimentary rocks by addition of abrasive nanoparticles was evaluated through ultrasonic cavitation experiments. Two kinds of typical sedimentary rocks, sandstone and organic-rich shale, were eroded by pure water cavitation and nanoparticle cavitation, respectively. Aluminum alloy was employed as a blank control group for contrast. The erosion rates, microstructure changes, and surface roughness before and after treatments were obtained to understand the erosion mechanisms. The results demonstrate that abrasive nanoparticle is capable of enhancing erosion and creating volume fragmentation on sedimentary rocks. In detail, the erosion rate on rock materials reaches ten times that on aluminum alloy. Mass loss of rock materials of the early stage is much severe compared to the erosion of the later stage. Brittle failure mode always dominates the rock failure that creates the volume fragmentation behavior. Enhanced cavitation intensity and high-speed abrasive impacts are the two main mechanisms to explain the enhanced erosion phenomenon. This work supports that cavitation abrasive waterjet with nanoparticles is a potential novel drilling and fracturing technology.