Using QCT, we made a longitudinal, population-based assessment of rates of bone loss over life at the distal radius, distal tibia, and lumbar spine. Cortical bone loss began in perimenopause in women and later in life in men. In contrast, trabecular bone loss began in young adulthood in both sexes. Although conventional wisdom holds that bone loss begins at menopause in women and later in life in men, this has not been examined longitudinally in population-based studies using precise technology capable of distinguishing cortical and trabecular bone. In an age- and sex-stratified population sample (n = 553), we measured volumetric BMD (vBMD) of trabecular and cortical bone by QCT annually for up to 3 yr at the distal radius (DR) and distal tibia (DT) (n = 552) and trabecular vBMD at baseline and 3 yr at the lumbar spine (LS) (n = 474). Substantial cortical bone loss began in middle life in women but began mainly after age 75 in men. In contrast, substantial trabecular bone loss began in young adult women and men at all three skeletal sites and continued throughout life with acceleration during perimenopause in women. Women experienced 37% and men experienced 42% of their total lifetime trabecular bone loss before age 50 compared with 6% and 15%, respectively, for cortical bone. Median rates of change in trabecular bone (%/yr) were -0.40, -0.24, and -1.61 in young adult women and -0.38, -0.40, and -0.84 in young adult men at the DR, DT, and LS, respectively (all p < 0.001). The early trabecular bone loss did not consistently correlate with putative causal factors, except for a trend with IGF-related variables at DT in women. However, in postmenopausal women and, to a lesser extent, in older men, higher rates of cortical and trabecular bone loss were associated with lower levels of biologically-active sex steroids and with higher levels of follicle-stimulating hormone and bone turnover markers. The late onset of cortical bone loss is temporally associated with sex steroid deficiency. However, the early-onset, substantial trabecular bone loss in both sexes during sex steroid sufficiency is unexplained and indicates that current paradigms on the pathogenesis of osteoporosis are incomplete.