To meet the needs of future wireless optical networks, this paper introduces a high-speed, hybrid multiplexed, coherent free-space optical (FSO) communication system that integrates an orbital angular momentum (OAM) multiplexed signal with an orthogonal frequency division multiplexing (OFDM) technique. Two independent QAM polarized beams, each carrying in-phase and quadrature (I/Q) phase 16-QAM-OFDM modulated data, are combined using mode division multiplexing (MDM) to increase the capacity of the proposed system. The reason of choosing OFDM is its capability to support higher data rate, and mitigating intersymbol interference (ISI). The signal is detected using a coherent detection-based digital signal processing (DSP) algorithm at the receiver end. The proposed hybrid FSO system is evaluated in low and heavy dust environments using bit error rate (BER), link distance, optical signal-to-noise ratio (OSNR), and received optical power performance matrices. The simulation results demonstrate the successful transmission of a 120 Gb/s single carrier over the longest link ranges of 1.5 and 0.40 km, respectively, under low and heavy dust weather environments below the signal degradation threshold value (forward error correction (FEC) limit) of BER 2.2 × 10–3 in strong turbulent conditions.