Abstract This paper proposes a novel wavelet-based method for the computation of the milling stability and surface location error (SLE). The governing equation of milling processes considering the self-vibration and forced-vibration is formulated as a delayed differential equation with state space form. The total time period is considered as a whole, sampled by the Chebyshev nodes. Then, the vibration displacement corresponding to each sampling grid point is fitted by a weighted linear sum of Gegenbauer orthogonal polynomials. A new method named direct differentiation based algorithm (DDBA) is proposed to determine these weight coefficients, which gets rid of dependence on the classic differential operator. On the one hand, the Floquet matrix is constructed to calculate the milling stability. On the other hand, the coefficient matrix independent of time is established to predict the SLE. The benchmark and experimentally verified examples are employed to validate the effectiveness of the proposed approach. The optimal combination of orthogonal polynomials is fixed by exploring the effect of the control parameter on the predictive accuracy. At last, the calculation vulnerability that can cause the prediction accuracy to be dropped obviously in some step lengths is revealed and repaired.