Body height is associated with environmental conditions. It has been suggested that under poor conditions when inequality within a population increases, also the variability in height tends to increase. We studied the association of body height, within-country variability in height and geographic and historic origin in 767 growth studies carried out in 80 countries, published between 1794 and 2013, with data on N = 78,184 infants age 2 years, and N = 2,130,729 juveniles age 7 years. The studies represent almost the whole spectrum of economic diversity in human societies since the end-18(th) century. 207 studies contained data for both infants and juveniles with 50,819 subjects (age 2), and 123,078 subjects (age 7). Multiple linear regressions showed significant interactions between height, sex, historic year of the study, geographic origin, and within-study standard deviation for height with multiple R-squared = 0.527, p < 0.001, at age 2, and multiple R-squared = 0.436, p < 0.001, at age 7. Yet, the two age groups differed in respect to within-study standard deviation for height. We found a significant association between body height and within-study standard deviation for height only at age 2: tall infant populations are less variable in height (r = -0.27, p < 0.01). There was no such association in children aged 7 years. Tall children from affluent and short children from less affluent countries do not differ in the variability of body height. The data suggest that the 'environmental adversity' hypothesis for variation in growth: small mean values for height go along with large standard deviations for height, does not apply for children at age 7.