ABSTRACTThe ever‐augmenting demand of data channel‐bandwidth by the end users has resulted in the exploration of new transmission techniques for wireless transmission networks. In this work, we have demonstrated the modeling of a single‐channel free space optics (FSO) transmission system by employing polarization‐division‐multiplexing (PDM) technique. Further, in‐phase‐quadrature modulation (IQM) has been used to increase the transmission efficiency of the system. To decrease the baud‐rate of the system, we have proposed the use of 64‐level quadrature amplitude modulation (QAM) signals. The net transmission rate of the system is 120 Gbps with a baud‐rate of 10 Gbps. Furthermore, the overcome the signal degradation of the 120 Gbps signal propagating through atmospheric channel, we have reported the use of advanced digital signal processing (DSP) algorithms. The system's performance is investigated for adverse external climate weather conditions and the results reported show reliable 120 Gbps data transmission along range between 870 m and 16.25 km with good performance of BER and EVM 7.5%.