Brittle materials such as glass and ceramics have fragile physical properties that result in poor stability in the machining process. Ultrasonic vibration-assisted polishing (UVAP) can improve the surface quality and machining efficiency of the workpiece, which is beneficial to solve the machining problems of brittle materials. In this paper, a theoretical model is established to predict the material removal rate (MRR), surface roughness, maximum contact radius, and maximum material removal depth of BK7 optical glass for different machining parameters. The new model mainly considers the following aspects: (1) Considering the randomness of the shape, the abrasive particle is modeled as a truncated polyhedron. (2) Due to the randomness of distribution, the abrasive particles are considered to be randomly distributed in the contact area between the polishing tool and the workpiece. (3) Based on the Preston equation, a material removal model is established which considers the micro-contact state of the abrasive particles with the workpiece. Compared with the experimental results of BK7 optical glass, the new model has high accuracy in predicting machining performance. It is meaningful to understand the mechanism of UVAP and improve the machining efficiency.