Preterm birth adversely affects postnatal brain development. In order to investigate the critical gestational age at birth (GAB) that alters the developmental trajectory of gray and white matter structures in the brain, we investigated diffusion tensor and quantitative T2 mapping data in 43 term-born and 43 preterm-born infants. A novel multivariate linear model—the change point model, was applied to detect change points in fractional anisotropy, mean diffusivity, and T2 relaxation time. Change points captured the “critical” GAB value associated with a change in the linear relation between GAB and MRI measures. The analysis was performed in 126 regions across the whole brain using an atlas-based image quantification approach to investigate the spatial pattern of the critical GAB. Our results demonstrate that the critical GABs are region- and modality-specific, generally following a central-to-peripheral and bottom-to-top order of structural development. This study may offer unique insights into the postnatal neurological development associated with differential degrees of preterm birth.