With the arrival of the 5th generation mobile network, the number of user devices is increasing exponentially, and thus it is necessary to expand the capacity of transmission systems. In order to further improve the system spectral efficiency on the basis of existing mobile fronthaul devices, we propose a hybrid digital-analog fronthaul transmission system with adaptive insertion of analog bandwidth, which can dynamically change the position of inserted analog bandwidth based on the state information of free space optical (FSO) channel. We consider the effects of atmospheric attenuation and turbulence on the FSO channel and derive an analytical expression for the maximum analog signal bandwidth that can be inserted into the first null of the digital signal spectrum to meet BER requirement of 3.8 × 10-3. Through a comprehensive simulation, it is verified that the analog bandwidth is obtained by this expression can exactly represent the lower bound of the simulation results under weak turbulence condition. The obtained results show that the maximum insertable analog bandwidth beyond the spectral null of the digital signal can reach 10% of the digital signal bandwidth, even in the FSO link with a transmission distance of 0.5 km and attenuation factor of 8 dB/km.