The heterogeneity of mass concrete often leads to the initiation of micro-cracks in gravity dams, while the accumulation of damage of concrete on the meso-scale leads to the cracking behaviour on the macro-scale. To improve the seismic safety of the dam, a comprehensive study on the effect of heterogeneous properties is crucial in seismic fragility analysis. A novel methodology considering concrete heterogeneity for seismic fragility analysis was developed by combining damage element model for seismic failure analysis, a five-level standard for failure modes-based seismic damage state and a method to generate seismic fragility curves. Then, seismic failure analysis of the Koyna gravity dam was taken as an example to verify the correctness of the damage element model, the damage process of dam from micro-cracks to macro-cracks were simulated. Finally, the seismic fragility of a practical dam in China was studied to verify the effectiveness of the proposed methodology. The results showed that simulated failure modes-based fragility curves were suitable for the target dam and more capable of considering the uncertainty of concrete material parameters. The obtained fragility curve proved valuable in optimizing seismic design, reinforcement and maintenance measures and improve seismic capability of the dam.