The current image processing techniques heavily rely on classic windows for suppressing spectral leakage. However, the limited capability of these classic windows to suppress spectral leakage restricts their application in the field of high-precision measurement. This paper proposes a novel family of windows capable of suppressing spectral leakage associated with the Moiré fringes pattern, thereby enhancing the accuracy of lithography alignment. Specifically, a novel two-dimensional Hanning self-convolution window is introduced, which is derived from the Hanning window to inherit its advantage of ultra-rapid sidelobe decay. The behaviors of the main lobe and desirable sidelobes of the 1st- to 4th-order are examined. Additionally, a novel fringe phase extraction algorithm is established based on the proposed window, which demonstrates high accuracy and efficiency. Simulation and experimental results validate the feasibility and practicality of the proposed window. In particular, the proposed window outperforms classic windows in suppressing spectral leakage, thereby successfully enhancing the phase extraction accuracy to achieve high-precision misalignment measurement at the sub-2-nm level (1.47 nm @ 3σ criterion). The proposed method can be easily extended to other real engineering applications utilizing fringe to achieve high-precision measurements, such as fringe projection profilometry, damage detection, and visual mechanical vibration monitoring.