The current study aimed to explore the combined and individual effects of vitamin D (VitD) status in three trimesters during pregnancy and cord blood (CB) on child growth trajectories from birth to 4 years of age. Pregnant women (n = 1100) were recruited between 2013 and 2016 in the Shanghai Birth Cohort (SBC) Study. A total of 959 mother-child dyads were included. VitD status was measured by LC-MS/MS at three trimesters (T1, T2, T3) and CB. Children's weight, length/height, and head circumference were assessed at birth, 42 days, 6, 12, 24 months, and 4 years of age, and standardized into z-scores [weight-for-age z-score (WAZ), length-for-age z-score (LAZ), head circumference-for-age z-score (HCZ) and weight-for-length z-score (WLZ)]. Using the group-based trajectory model (GBTM), the trajectories of the four growth parameters were categorized into discrete groups. The generalized estimating equation (GEE) was employed to analyze the mixed effect of 25(OH)D throughout pregnancy on growth trajectories. The association between 25(OH)D status and each growth trajectory group was examined by multivariable logistic regression. Each 10 ng/mL increase in 25(OH) throughout three trimesters was not associated with four anthropometric parameters. Each 10 ng/mL increase in VitD in T3 was associated with a lower risk in the WAZ high-increasing trajectory (aOR: 0.75; 95% CI: 0.62, 0.91; p < 0.01). Each 10 ng/mL increase in VitD in CB was associated with a lower risk in the WAZ high-increasing trajectory (aOR: 0.57; 95% CI: 0.43, 0.76; p < 0.01). No significant association was found between maternal or CB VitD and LAZ or HCZ. Three trimesters' VitD throughout pregnancy had no persistent effect on the offspring's growth trajectory. However, higher VitD status in the third trimester and CB related to a lower risk of high-increasing WAZ from birth to 4 years of age. Elevated VitD levels in late pregnancy and cord blood may protect against continuous early-life weight growth at high levels.