Emerging evidence indicates that ambient particulate matter (PM) is harmful to male fertility, but the associations between ambient PM exposure and semen quality were inconsistent. This study aimed to quantitatively evaluate the association between ambient PM exposure and semen quality based on a large prospective cohort. Using data from the prospective assisted reproduction cohort in Anhui province, China, we included 15,112 males with 28,267 semen tests, whose partner has undergone assisted reproductive technology from September 1, 2015 to September, 22 2020. Individual ambient PM, gaseous air pollutants, and temperature exposures of the participants during 0–90, 0–9, 10–14, and 70–90 days before semen quality tests were evaluated using inverse distance weighting interpolation. Linear mixed-effects models were conducted to evaluate the relationship between PM2.5 and PM10 exposures and standardized semen quality parameters. Models were adjusted for age, body mass index, smoking, drinking, education attainment, occupation type, sampling month, temperature and the principal component of gaseous air pollutants. PM2.5 and PM10 were inversely associated with sperm concentration, total sperm count, total motility, progressive motility, total motile sperm count, and progressively motile sperm count during 0–90, 0–9, and 70–90 days period (all p < 0.05), but not 10–14 days period. The regression coefficients of PM2.5 exposure on semen quality parameters during 0–90 days period were larger than 0–9 and 70–90 days periods, and the effects of PM2.5 on semen quality parameters were stronger than PM10. Our results showed that ambient PM2.5 and PM10 exposures were associated with semen quality, during 70–90 days and 0–9 days before sampling, and the entire spermatogenesis process. The effects of PM2.5 on semen quality parameters were stronger than PM10, and the long-term effects of PM2.5 and PM10, throughout spermatogenesis, were stronger than the short-term effects.