In this work, we investigate the performance comparison of different modulation schemes viz. non return-to-zero (NRZ), alternate mark inversion (AMI), return-to-zero differential phase shift keying (RZ-DPSK), and non return-to-zero differential phase shift keying (NRZ-DPSK) in a 2 × 40 Gbps–40 GHz Radio over free space optics (RoFSO) transmission link incorporating mode division multiplexing (MDM) of two distinct Hermite Gaussian (HG) modes (HG00 and HG01) under the influence of different atmospheric turbulence conditions. Further, the effect of increasing receiver antenna aperture diameter on the link performance is investigated and compared for all four modulation schemes under different turbulence conditions. It is observed that as the strength of the atmospheric turbulence increases, the performance of the link degrades whereas the Bit error rate and the Quality Factor (Q Factor) of the received signal are significantly improved by increasing the receiver aperture diameter. Among the four modulation schemes, link with NRZ-DPSK modulation scheme gives the best performance under the influence of atmospheric turbulence followed by RZ-DPSK, AMI, and NRZ scheme. Also, the proposed NRZ-DPSK scheme based MDM-RoFSO link is investigated under different weather conditions and the results are compared with the previously reported works in the literature.