Optical surface scattering analyses based on diffractive optics (DO) are typically applied to one surface; however, there is a need for simulating surface scattering losses for devices having many surface interactions such as light pipes. Light pipes are often simulated with geometric optics (GO) using ray tracing, where surface scattering is driven by the surface slope distribution. In the DO case, surface scattering analyses depend on the spatial frequency distribution and amplitude as well as wavelength, with the sinusoidal grating as a fundamental basis. A better understanding of the link, or transition, between DO and GO scattering domains would be helpful for efficiently incorporating scattering loss analyses into ray trace simulations. A formula for the root-mean-square (rms) scattered angle width of a sinusoidal reflection grating that depends only on the surface rms slope is derived from the nonparaxial scalar diffraction theory, thereby linking it to GO. The scatter angle’s mean and rms width are evaluated over a range of grating amplitudes and periods using scalar theory and full vector simulations from the COMSOL® wave optic module for a sinusoidal reflection grating. The conditions under which the diffraction-based solution closely approximates the GO solution, as predicted by the rms slope, are identified. Close agreement is shown between the DO and GO solutions for the same surface rms slope scattering loss due to angular filtering near the critical angle of a total internal reflection (TIR) glass-to-air interface.