The image-type angular displacement measurement method based on linear image recognition has garnered attention because of its higher frequency response, strong fault tolerance, and high robustness. In the small-size image angular displacement measuring device, due to the limited pixel size, the circular grating cannot place more lines when realizing single-channel absolute coding recognition on a small diameter grating disk. This leads to larger errors in angular displacement measurement when the line density of the grating disk is low. To improve the measurement accuracy of small-scale displacement, the present work aimed to reduce this error by studying the error compensation method involving low-density grating disks. First, the mechanism of linear image-type angular displacement measurements is described, and a measurement algorithm based on linear scan images is proposed. Second, the measurement error model for low-density grating disks is established according to the proposed measurement algorithm. Third, a simplified error compensation algorithm based on a harmonic model is developed. Finally, simulations and experiments are performed to verify the performance of the proposed algorithm. Simulation results show that the developed harmonic compensation algorithm can effectively reduce the error caused by the low-density circular grating. When the proposed error compensation algorithm is applied to a grating disk with a 62 mm diameter and 2 <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><sup>N</sup></i> lines, the measurement accuracy is improved from 8.14” to 4.78”. The proposed error compensation algorithm can significantly improve the accuracy of linear image-type angular displacement measurements involving low-density grating disks, and the results presented herein laid a foundation for improving the accuracy and engineering applicability of angular displacement measurement technology.