In the ultra-precision machining process, it is important to detect the tool-workpiece distance by the laser diffraction effect. However, there is no absolute ideal situation for the tool setting of laser diffraction. It is necessary to consider the influence of installation tilt error on the tool setting accuracy and its compensation. In this paper, referring to the ideal optical axis, the influences of the incident laser tilt, CCD tilt, and rectangular orifice tilt on the peak position of the diffraction fringes in the CCD phase are modeled and analyzed to determine the relative effects on the tool setting accuracy for tool-workpiece distance detection. The tilt angle of the incident laser is measured by extracting the CCD pixel position, where the central peak point is located. The CCD tilt angle is detected by extracting the CCD pixel positions, where the first-degree peak points are located. The inclination direction and the inclination angle of the rectangular orifice are detected by extracting the bending direction and the bending degree of the fringes in the horizontal direction, and the error compensation is performed according to the inclination angle. This study establishes a compensation method for installing tilt error, and the optimized installation process parameters can be acquired under certain experimental conditions according to the experimental data to ensure the results' accuracy.