Small-for-gestational-age (SGA) and large-for-gestational-age (LGA) births are major adverse birth outcomes related to newborn health. In contrast, the association between ambient air pollution levels and SGA or LGA births has not been investigated in Japan; hence, the purpose of our study is to investigate this association. We used birth data from Vital Statistics in Japan from 2017 to 2021 and municipality-level data on air pollutants, including nitrogen dioxide (NO2), sulfur dioxide (SO2), photochemical oxidants, and particulate matter 2.5 (PM2.5). Ambient air pollution levels throughout the first, second, and third trimesters, as well as the whole pregnancy, were calculated for each birth. The association between SGA/LGA and ambient levels of the air pollutants was investigated using crude and adjusted log-binomial regression models. In addition, a regression model with spline functions was also used to detect the non-linear association. We analyzed data from 2,434,217 births. Adjusted regression analyses revealed statistically significant and positive associations between SGA birth and SO2 level, regardless of the exposure period. Specifically, the risk ratio for average SO2 values throughout the whole pregnancy was 1.014 (95% confidence interval [CI] 1.009, 1.019) per 1ppb increase. In addition, regression analysis with spline functions indicated that an increase in risk ratio for SGA birth depending on SO2 level was linear. Furthermore, statistically significant and negative associations were observed between LGA birth and SO2 except for the third trimester. It was suggested that ambient level of SO2 during the pregnancy term is a risk factor for SGA birth in Japan.