A new method of optical signal-to-noise ratio (OSNR) monitoring in presence of fiber nonlinearities, suitable for Nyquist Wavelength Division Multiplexing (Nyquist-WDM) coherent optical transmission system, is proposed, in which the correlation function (CF) between the symbols who come from neighboring channels is used. The effectiveness of the proposed method is numerically verified in a 28 Gbaud per channel Nyquist-QPSK-WDM coherent system. The results show that, under the configuration with fiber nonlinearities, this method can achieve OSNR monitoring with the error less than 1.5dB in the OSNR range of 10–24dB, and keep valid regardless of the transmission distances, channel input powers and number of WDM channels.