A noble theoretical approach is presented to evaluate the bit error rate (BER) performance of an optical polarization division multiplexed (PDM) 4-multilevel quadrature amplitude modulation (4-QAM) transmission system under the combined influence of polarization mode dispersion (PMD) and group velocity dispersion (GVD) in a single mode fiber (SMF). The analysis is carried out considering a coherent homodyne receiver. The exact probability density function (pdf) fluctuation due to PMD and GVD is evaluated from its moments using a Monte-Carlo simulation technique. Average BER is evaluated by averaging the conditional BER over the pdf of the random phase fluctuation. BER performance results are evaluated for different system parameters. It is found that PDM 4-QAM coherent homodyne system doubles the data rate but suffers more power penalty than the 4-QAM system. Results show that for a BER of 10<sup>-9</sup> at DGD of 0.5T and GVD value of 1700 ps/nm the PDM 4-QAM system needs 2.21 dB more power than 4-QAM systems giving the leverage of doubling the data rate.