The intricate dynamic responses of structures, particularly when confronted with uncertainty, poses significant challenges in constructing reliable analytical models, thus resulting in unreliable fragility assessment. This study presents an approach that integrates data-driven techniques with real-time hybrid simulation (RTHS) to achieve reliable fragility analysis. RTHS data and imprecise surrogate models are fused using a multi-fidelity Monte Carlo predictor (MFMC), yielding unbiased estimations for parameters of fragility curve. The MFMC predictor can be considered as an optimized forecasting tool that harnesses the computational efficiency of low-fidelity simulations and the precision of high-fidelity experiments. RTHS is utilized for obtaining high-fidelity experimental data, part of which is leveraged to establish low-fidelity surrogate models, ultimately enabling multi-fidelity analysis through direct data-to-data analysis. A two-story steel moment-resisting frame equipped with self-centering viscous dampers is used as a proof-of-concept to demonstrate the proposed method. Structural and ground motion uncertainties are incorporated to reflect practical engineering environment. The low-fidelity Kriging models were built using portion of high-fidelity RTHS data. Simulation and experimental investigations were conducted to evaluate the effectiveness and efficiency of the proposed approach. The results suggest that the proposed method provides a promising tool for accurately evaluating seismic fragility, particularly when simulation-based methods are not feasible.