Abstract Fly cutting is widely used in manufacturing of large-scale, high-precision optical components. However, the discontinuity of fly cutting machining leads to significant relative vibrations between the tool and the workpiece. The cutting process generates periodic waves along the cutting direction, which will deteriorate the wavefront characteristics of optical components. Based on the machining dynamics, this paper proposes a direct integration method to predict the waviness error of the machined surface. The cutting force model of fly cutting is established. The multi-mode characteristics of the spindle-tool system are measured by the experimental method. Then, the influence of uncertainties on the calculation results is analyzed by the variance-based sensitivity analysis method. Finally, the plane cutting experiment verifies that the direct integration method effectively predicts the waviness error and its variation trend, and the waviness prediction research is important for optimization of the machining parameters.