The bit error rate (BER) of the M-QAM signal in analog/digital hybrid optical transmission (hybrid transmission) shows a floor-like feature due to impulse noise induced by clipping the subcarrier multiplexed signal at the laser threshold. This paper proposes a new BER analysis model, which is modified from our previous one based on Middleton's class A impulse noise model, and compares analytical results with experimental data. The analytical results agree particularly well with the experimental data for small optical modulation indexes (OMI) of the M-QAM signal; consequently, these results can be applied to estimating the bit error rate of the digital channel in hybrid transmission. There is also a tendency for the analysis results to diverge more greatly from the experimental data as the OMIs of the M-QAM signal increase. We also discuss the reasons for this divergence. © 1998 Scripta Technica. Electron Comm Jpn Pt 1, 81(8): 1–8, 1998