Abstract In this paper, the symbol-error rate (SER) performance of a coherent free-space optical (FSO) communication system in lognormal-Rician turbulence channel is analyzed using the generalized Gram-Charlier (GCC) series. We proposed the differential evolution (DE) algorithm to solve the parameters in GCC efficiently. It is shown that highly accurate closed-form SER expressions are obtained for M-ary phase-shift keying (MPSK) and M-ary quadrature amplitude modulation (MQAM) schemes with maximum ratio combining (MRC) technique. The asymptotic error rate analysis is presented to reveal the performance behavior in the high signal-to-noise (SNR) regime. The effects of imperfect phase noise compensation on the error rate performance are also studied, and it is found that the impact of phase compensation error can be small enough with loop SNR ρ c more than seven.