We report on the spectral characteristics of the phase noise enhancement of continuous-wave (CW) optical signals passing through a semiconductor optical amplifier (SOA) operating in the saturation regime and used for relative intensity noise (RIN) reduction. We show that the known effect of phase noise enhancement, attributed to the linewidth enhancement factor of the device, happens only at a limited band of the phase noise spectrum, and the actual measurable linewidth of the output CW signal may not be affected. While this phase noise enhancement is not shown as an increase of spectral linewidth, it can still affect system performance when coherent detection is used, especially in applications with relatively low symbol rates. Numerical simulations and experimental results are used to support the observation. A single spectral line from a quantum-dot mode-locked laser is used as the light source, which is known to have relatively high RIN (> −120 dB/Hz in the low frequency region). Experimental transmission of 16-QAM modulation with coherent detection has been performed at 5 GBd to assess the implication on system performance.