The image formation theory is presented for scanning differential heterodyne microscope respect to triangular-shaped grooves used for plasmon-polariton waveguiding. The scattering problem of the illuminating probe beams focused on the object surface is resolved with rigorous coupled-wave analysis (RCWA) for groove corrugated grating etched in a substrate with the finite conductivity. To adapt the RCWA method for SDHM the problems were considered associated with a discretization of the diffracted field and sampling the focused probe beams. The focused beams are expanded in terms of plane waves so that the each chosen incident wave is order of one another and then the diffraction problem is solved for every plane wave of TE polarization. The groove was decomposed into thin layers in which the permittivity and electromagnetic field are expanded in Fourier series. The diffraction of the probe beams by triangular groove was analyzed, at that the convergence of modulus and phase responses of SDHM in dependence of the number of the layers and the number of Fourier modes in optical field expansion was investigated. The elaborated algorithm was used for the simulation of differential responses from triangular grooves with various parameters.