The objective of this study was to examine the association between physical activity during childhood and adolescence and the risk of all-cause mortality in midlife. We analyzed data from a birth cohort (The 1958 National Child Development Survey), including births in England, Wales and Scotland. Physical activity was assessed using questionnaires at ages 7, 11 and 16. Death certificates defined all-cause mortality. Cumulative exposure, sensitive and critical periods, and physical activity trajectory from childhood to adolescence were tested using multivariate Cox proportional hazard models. The sweep the death was confirmed was defined as the time event. From age 23 to 55, 8.9% of participants (n = 9398) died. Physical activity in childhood and adolescence affected the risk of all-cause mortality in midlife. In men, physical activity at ages 11 [hazard ratio (HR): 0.77; 95% confidence interval (CI): 0.60-0.98] and 16 (HR: 0.60; 95% CI: 0.46-0.78) was associated with reduced risk of all-cause mortality. In women, physical activity at age 16 (HR: 0.68; 95% CI: 0.48-0.95) was associated with reduced risk of all-cause mortality. Physical activity in adolescence eliminated the risk of all-cause mortality associated with physical inactivity in adulthood in women. Physical activity during childhood and adolescence was associated with reduced risk of all-cause mortality with different effects by sex.