The associations of size at birth, with infant growth of head circumference, length and weight, and fat mass and body mass index in preschool children were examined. In a population-based prospective cohort study among 3,941 children, head circumference, length and weight until the age of 4 years were repeatedly measured. Catch-up and catch-down growth were defined as a change in standard deviation scores of >0.67 from birth to 2 years of age. Although most children born small and large size for gestational age showed infant catch-up and catch-down growth, respectively, their mean head circumference, length and weight remained smaller and larger respectively, until the age of 4 years. Catch-up growth in children with a small and appropriate weight for gestational age and lack of catch-down growth in children born with a large weight for gestational age were associated with higher body mass index in preschool children. Children born with an appropriate weight for gestational age with catch-up growth and children born with a large weight for gestational without catch-down growth had increased risks of childhood overweight {odds ratios: 3.11 (95% confidence interval[95% CI] 2.37, 4.08) and 12.46 (95% CI: 6.07, 25.58) respectively}. Children born small, appropriate and large size for gestational age have different growth patterns in early childhood and persistent differences in their head circumference, length, and weight until the age of 4 years. Children born with an appropriate weight for gestational age with catch-up growth and large weight for gestational children without catch-down growth have an increased risk of overweight.