Polarized low-coherence interferometry (PLCI) is a promising technology to absolute displacement measurement and sensor demodulation. To avoid jump errors induced by dispersion and achieve a high response, a dispersion compensation method based on fringe peak position estimation and polynomial fitting is proposed to get rid of the dependence on the priori model and birefringence parameters, and make this method very robust. The nearly linear resolution between the envelope and ${m}$ -order PLCI fringe (PLCIF) centers are found and verified with both Gauss and non-Gauss profile broadband light sources. Then, the zero-order PLCIF is estimated and located to demodulate the measurement displacement. The demodulated results are significantly influenced by dispersion of the birefringence optical wedge and then polynomial fittings of sensing curve are conducted to raise the sensing accuracy. In comparison to the priori model and parameters based method, the proposed method can effectively avoid jump errors and has a higher accuracy than the directly envelope center method. Experimental results indicate that the measurement accuracy is higher than 19.5 nm which is enhanced by 6.88 times, the resolution is better than 2 nm, and its processing data rate can reach 35 kHz on an embedded system.