Exposure to environmental pollutants during pregnancy may cause adverse birth outcomes in newborns. This study examined links between exposure to various residential environmental characteristics during pregnancy and newborn birth weight/head circumference, using the Taiwan Birth Cohort Study (TBCS) database with 14,829 subjects. We conducted simple and multiple linear regression analyses to find out the associations between residential environmental characteristics (molds, environmental tobacco smoke [ETS], household insecticide use, and home renovation or painting) and birth outcomes (birthweight, head circumference), and adjusting for demographics and environmental variables. The standardized beta coefficients were calculated. We also analyzed whether the associations varied by newborn sex and maternal body mass index (BMI). The regression analysis indicated that molds on multiple walls at the residence were significantly associated with lower birthweight (standardized β = −0.025, p = 0.003) and with smaller head circumference (standardized β = −0.023, p = 0.008) than no mold on walls after adjustment. Exposure to ETS did not result in significance for birthweight reduction but a significant reduction in head circumference (standardized β = −0.018, p = 0.047). The birthweight reductions associated with molds were −0.024 SD (standard deviation) for boys and −0.028 SD for girls, whereas the head circumference reduction with molds was sex-specific, being only significant for girls (standardized β = −0.029, p = 0.021). The analysis on BMI-separated data indicated that all significance occurred to mothers having normal BMI. Mold presence correlated significantly with reduced birth weight and head circumference, while ETS exposure only reduced head circumference. Girls and mothers with normal BMI were more affected by these reductions than boys or under/overweight mothers.