A novel signal processing method based on phase shift of reference signal is proposed for heterodyne interferometer. The integer fringe counting method based on overflow judgment and compensation can realize longtime and correct integer number measurement. In order to eliminate the influence of jitter in measurement signals on combination of integer and fraction fringe counting, the reference signal with phase shift of 180° is used to obtain integer compensating number to compensate the unstable integer number in unstable phase zone, which guarantees the correct combination of integer and fraction fringe counting. The principle of the proposed signal processing was described in detail. The static and dynamic resolution of the proposed method were discussed. A signal processing board based on FPGA was developed, and three tests were performed to verify the feasibility of the proposed method. A displacement measurement experimental setup was constructed, and two experiments verified the effectiveness of proposed method in application of an interferometer to realize precision displacement and testing of a stage.