The transmissibility-based operational modal analysis (TOMA) method is attracting more attentions due to its relaxation of assumptions about excitation properties, making accurately identifying the modes of actual engineering structures possible. However, in specific application on distributed vibration responses of huge hydraulic structures excited by broad-spectrum non-stationary earthquake, more proper selections of data segmentations on two dimensions (along time or spatial axis) are needed. Based on the concept of distributed vibration sensing optical fiber strain transmissibility, the operational modal analysis model based on single reference and poly references transmissibility under the strain format and the corresponding solutions of model and modal parameters are studied. A false mode elimination method combined with the continuity of spatial distribution of optical fiber measurements, a response sequence set optimization method that comprehensively consider amplitude/PSD non-stationarity, and the principle about how to select (non) reference points (sets) considering the spatial distribution of measurement signal-to-ratio are further developed. The case study shows that the proposed combined method could improve the modal parameter identification accuracy of concrete dams under broad-spectrum non-stationary excitation, and the distributed optical fiber vibration sensing technology can provide a rich (non) reference point (set) selection combination for this method.