To improve the robustness of frequency-modulated continuous-wave (FMCW) distance measurement systems, we clarified the effect of temperature changes in the surroundings. In particular, the temperature change causes a phase change between the slow-axis and fast-axis polarization of the optical fiber, which in turn causes a distance error. Numerical analysis using a model of the phase state of polarization revealed that a distance gain error occurs in the reference fiber interference unit and a distance offset error occurs in the measurement fiber interference unit of the FMCW distance measurement system. We propose two measurement-error reduction methods that suppress the phase change in polarization in accordance with the characteristics of each interference unit. Experimental evaluations showed that the measurement errors were reduced from 5 μm to 1 μm and the temperature robustness of the system was improved.