Polarization-sensitive optical coherence tomography (PS-OCT) enables three-dimensional imaging of biological tissues based on the inherent contrast provided by scattering and polarization properties. In fibrous tissue such as the white matter of the brain, PS-OCT allows quantitative mapping of tissue birefringence. For the popular PS-OCT layout using a single circular input state, birefringence measurements are based on a straight-forward evaluation of phase retardation data. However, the accuracy of these measurements strongly depends on the signal-to-noise ratio (SNR) and is prone to mapping artifacts when the SNR is low. Here we present a simple yet effective approach for improving the accuracy of PS-OCT phase retardation and birefringence measurements. By performing a noise bias correction of the detected OCT signal amplitudes, the impact of the noise floor on retardation measurements can be markedly reduced. We present simulation data to illustrate the influence of the noise bias correction on phase retardation measurements and support our analysis with real-world PS-OCT image data.