The challenging requirements of large scale quantum information processing using parametric heralded single photon sources involve maximizing the interference visibility while maintaining an acceptable photon generation rate. By developing a general theoretical framework that allows us to include a large number of spatial and spectral modes together with linear and non-linear optical elements, we investigate the combined effects of spectral and photon number impurity on the measured Hong–Ou–Mandel interference visibility of parametric photon sources, considering both threshold and number resolving detectors, together with the effects of spectral filtering. We find that for any degree of spectral impurity, increasing the photon generation rate necessarily decreases the interference visibility, even when using number resolving detection. While tight spectral filtering can be used to enforce spectral purity and increased interference visibility at low powers, we find that the induced photon number impurity results in a decreasing interference visibility and heralding efficiency with pump power, while the maximum generation rate is also reduced.