Uncertainty in the seismic fragility assessment framework involves two basic sources including irreducible aleatoric and inherent epistemic uncertainties. The aleatoric uncertainty is considered to capture the uncertainty in response due to variation in ground motion properties, whereas the epistemic uncertainty stems from the variation in material, geometric, and loading properties. The effect of these uncertainties on seismic fragility estimates of reinforced concrete (RC) buildings is investigated in the study. Three configurations of RC building frames with and without masonry infill walls are examined - bare, open ground storey, and fully infilled frames. Samples of structural inputs related to material, geometry, and loading are generated to estimate the epistemic uncertainty. More than 100 ground motions, both recorded as well as synthetic, are considered for determining the aleatoric uncertainty. Although a few documents recommend the effective uncertainty values for fragility assessment of RC buildings, the present study shows that such values have limitations when masonry infill walls are considered. More specifically, recommendations on the uncertainty associated with open ground storey buildings, which have severe vertical irregularities due to the absence of masonry infill walls from the ground storey, are not plenty in the literature. Effectiveness of considering uncertainty in only some sensitive input parameters on overall uncertainty is studied, and recommendations made for uncertainty values that can be considered for masonry infill RC frame buildings with different infill configurations based on results of thousands of nonlinear time history analyses. Different uncertainties are presented as a continuous function of seismic intensity measure to facilitate the estimation of seismic fragility.