In this paper, we propose an in-phase/quadrature (IQ) imbalance estimation scheme for mixed transmitter (Tx) and receiver (Rx) IQ imbalance in the optical coherent transmission system. The proposed scheme is based on traditional IQ imbalance compensation algorithm Gram-Schmidt orthogonalization procedure (GSOP) and the proposed geometric construction method (GCM). In the actual transmission system, frequency offset (FO) and phase noise (PN) are recognized as part of received signal, which has an effect on the estimation of IQ imbalance. At the beginning of the proposed scheme, the GSOP algorithm is used to estimate and compensate Rx IQ imbalance according to the statistical characteristics of the received signal, then standard frequency offset compensation algorithm and k-means clustering aided blind phase search (BPS) algorithm are used to compensate FO and PN respectively. Finally, the proposed method GCM is used to estimate Tx IQ imbalance. 34 Gbaud QPSK and 16QAM signals are experimentally performed to demonstrate the proposed IQ imbalance estimation scheme, the results show the proposed scheme has accurate estimation capability with low estimation algorithm complexity.