The accuracy of estimated modal parameters using different output-only system identification techniques is dependent on the amplitude of the structural response and the associated measurement noise. Unlike, a typical experiment using attached sensors and a multi-channel data acquisition (DAQ) system, optical measurement systems can have high levels of measurement noise that may result in estimated modal parameters that have low accuracy. Although the BSS-based modal parameter estimation techniques are able to identify modal characteristics of the structures from system responses, the performance of these techniques are generally limited by the damping values of structures and the number of sensors mounted on the structure. This paper proposes a new hybrid output-only system identification method that improves the extraction of modal parameters from optically measured system responses. The proposed Operational Modal Analysis (OMA) method employs a BSS learning technique, referred to as Complexity Pursuit (CP), to recover the source signals (i.e. time-domain modal responses). Afterwards, the recovered time-domain modal responses were further processed using the poly-reference least square complex frequency domain (PLSCF) technique to accurately estimate modal information from the recovered source signals. In this work, the numerical response of a cantilevered beam is processed using the hybrid output-only system identification method to evaluate the accuracy of the proposed technique in identifying modal parameters in the presence of noise. Furthermore, the proposed method is used to characterize the optically measured dynamic response of a 2.3-m long wind turbine blade. The obtained results indicate that the proposed output-only system identification is robust to the measurement noise, and since the contribution of the other modal responses are attenuated after applying BSS the hybrid methodology has a potential to identify accurate dynamic characteristics of structures from optically measured data.