Abstract To enhance the transmission capacity for energy consumption and low cost system, mode division multiplexing (MDM) using mode 0 and mode 1 is proposed and investigated. A pair of eight downlink and uplink wavelengths is transmitted by using each MDM mode in an integrated bidirectional next generation passive optical network and free space optics (NGPON–FSO) system at 10 Gbps per channel transmission rate under the impact of FSO–fiber links impairments. Simulation results indicate that received power of −21.2 dBm with 0.2 dB power penalty can be obtained over 1 km FSO and 100 m multimode fiber under unfavourable turbulent effects. The system provides extended FSO link range, and fiber range of 150 km and 600 m respectively at threshold limit of 10−3. Besides, the mathematical analysis depicts the receiver sensitivity and splitter power budget of 42 dBm and 18 dB, respectively, for the proposed system. The system offers finest performance than other pre-existing systems.