Abstract We study two-sector R&D model with endogenous human capital accumulation. Allowing for fractional human capital spillover from parents to their offspring, which are subject to congestion in fertility rate, we establish non-monotonic relations between population growth and economic growth. These non-monotonic relations, which are polynomial in general, are determined by the base level of human capital spillover and the magnitude of the congestion effect: a U shape relation can arise under low congestion factor, whereas a hump shape may present for high congestion factor. Hence this work contributes to the recent line of modified R&D-based growth models, aimed to align theory with the mixed empirical data on the relation between population growth and economic prosperity.