Polarization navigation techniques become more and more significant for a wide range of researches and applications. However, most of the polarization navigation sensors can only output the navigation angle information of a carrier based on the angle of polarization (AoP) from the skylight in the daytime; they cannot provide the displacement information in day and night. In this paper, a polarized contrast-enhanced (PCE) optical flow method is proposed to improve the accuracy of displacement estimation and reduce the influence of the ambient brightness. We acquire the degree of linear polarization (DoLP) images during the motion of the carrier in low illumination environment, and the hypothesis of the DoLP continuity is used for the theoretical foundation of this PCE optical flow method under the conditions of parallel light source and small displacements. The displacement calibration on the experimental platform is obtained by using a laser interferometer and spectrometer, and the contrast of the polarimetric image and the calculated polarized optical flow value are used as the validation criteria. Under 0.7198Lx illumination intensity, the cumulative error of the displacement obtained by the PCE method is less than 8 mm and the accuracy of the displacement estimation is almost 2 times higher than that of the traditional Lucas-Kanade(LK) optical flow method. Our experimental results have also shown that under 10°, 10−1, or even 10-2 levels of the standard illumination, the displacement values from the PCE optical flow method are in good agreement with the theoretical values.