The machining induced edge chipping of hole manufacturing restricts the productivity and design capability of products made from brittle materials. Rotary ultrasonic machining (RUM) has been sufficiently proved as a suitable approach for hole drilling in brittle materials with reduced cutting force and improved hole edge quality. However, apparent edge chipping is still an important restriction of RUM application. In this study, a compound step-taper tool for RUM was designed to further reduce the edge chipping defects via tool design. RUM tests on quartz glass were conducted to evaluate the effectiveness of the compound step-taper tool. The experimental results demonstrated that the compound tool could reduce the edge chipping size at the hole exit and entrance by 60%–80% and 35%–50%, respectively. The mechanism of edge chipping reduction at the hole exit is that cutting force decreased as the undrilled thickness decreased. This is due to the wedge-type contact structure between the tool’s taper face and the workpiece material. Furthermore, the edge chipping reduction at the hole entrance occurs because of the shielding effect of residual cracks. Residual cracks produced by the second outermost abrasives of the tool’s taper face suppressed the initiation and propagation of lateral cracks that were produced by the outermost abrasives of the tool’s taper face. Theoretical analysis indicated that the compound tool step is beneficial for guaranteeing the effectiveness of tool’s taper face.