In this paper, a novel compensation scheme for receiver (Rx) in-phase/quadrature (IQ) mismatch is proposed in coherent optical orthogonal frequency division multiplexing (CO-OFDM) system in the presence of laser phase noise. In this scheme, laser phase noise and channel distortion were combined as a new channel transfer factor, the IQ mismatch factor and initial channel transfer factor can be estimated independently based on the relationship of IQ mismatch factors. And the channel transfer factor can be updated on a symbol-by-symbol basis which retrieves an estimation of the phase noise value by extracting and averaging the phase drift of all OFDM sub-channels. Numerical results indicate that when the phase and amplitude mismatch are 10° and 2dB respectively, a 1.6dB optical signal-to noise ratio is improved at laser linewidth of 60kHz. Furthermore, the complexity of the proposed method is analyzed in terms of the number of required complex multiplications per bit.