The present paper is concerned with the Rayleigh wave propagation in the machined surface with a thin damaged layer with residual stress. We assume that the layer and the half-space are bonded perfectly to each other. Biot’s theory of small deformations influenced by initial stress forms the basis for this study. With the help of the effective boundary condition method, a third-order approximate secular equation of Rayleigh waves is established for the case that the layer and half-space are both orthotropic. In addition, an explicit third-order approximate formula for the Rayleigh wave velocity is derived from the secular equation. By considering a sample of silicon wafer by fine grinding with fine abrasive grains, the accuracy of the approximate formula has been verified. This study is meant to serve as the mathematical foundation for non-destructive testing of residual stress in the practical applications.