Identification of individual-level barriers associated with decreased activity in older age is essential to inform effective strategies for preventing the health outcomes associated with high sedentary behavior and lack of physical activity during aging. To assess cross-sectional and prospective associations of a large set of factors with objectively assessed sedentary time and physical activity at older age. This population-based cohort study was conducted among participants in the Whitehall II accelerometer substudy with accelerometer data assessed in 2012 to 2013. Among 4880 participants invited to the accelerometer substudy, 4006 individuals had valid accelerometer data. Among them, 3808 participants also had factors assessed in 1991 to 1993 (mean [SD] follow-up time, 20.3 [0.5] years), 3782 participants had factors assessed in 2002 to 2004 (mean [SD] follow-up time, 9.1 [0.3] years), and 3896 participants had factors assessed in 2012 to 2013 (mean follow up time, 0 years). Data were analyzed from May 2020 through July 2021. Sociodemographic factors (ie, age, sex, race and ethnicity, occupational position, and marital status), behavioral factors (ie, smoking, alcohol intake, and fruit and vegetable intake), and health-related factors (ie, body mass index, 36-Item Short Form Health Survey (SF-36) physical and mental component summary scores [PCS and MCS], and number of chronic conditions) were assessed among 3808 individuals in 1991 to 1993; 3782 individuals in 2002 to 2004; and 3896 individuals in 2012 to 2013. High alcohol intake was defined as more than 14 units of alcohol per week, and high fruit and vegetable intake was defined as twice daily or more. Accelerometer-assessed time spent in sedentary behavior, light-intensity physical activity (LIPA), and moderate to vigorous physical activity (MVPA) in 2012 to 2013 were analyzed in 2021 using multivariate linear regressions. A total of 3896 participants (986 [25.3%] women; age range, 60-83 years; mean [SD] age, 69.4 [5.7] years) had accelerometer data and exposure factors available in 2012 to 2013. Older age, not being married or cohabiting, having overweight, having obesity, more chronic conditions, and poorer SF-36 PCS, assessed in midlife or later life, were associated with increased sedentary time at the expense of time in physical activity. Mean time differences ranged from 9.8 min/d (95% CI, 4.1 to 15.6 min/d) of sedentary behavior per 10-point decrease in SF-36 PCS to 51.4 min/d (95% CI, 37.2 to65.7 min/d) of sedentary behavior for obesity vs reference range weight, from -6.2 min/d (95% CI, -8.4 to -4.1 min/d) of LIPA per 5 years of age to -28.0 min/d (95% CI, -38.6 to -17.4 min/d) of LIPA for obesity vs reference range weight, and from -5.3 min/d (95% CI, -8.2 to -2.4 min/d) of MVPA per new chronic condition to -23.4 min/d (95% CI, -29.2 to -17.6 min/d) of MVPA for obesity vs reference range weight in 20-year prospective analyses for men. There was also evidence of clustering of behavioral factors: high alcohol intake, high fruit and vegetable consumption, and no current smoking were associated with decreased sedentary time (mean time difference in cross-sectional analysis in men: -12.7 min/d [95% CI, -19.8 to -5.5 min/d]; -6.0 min/d [95% CI, -12.3 to -0.2]; and -37.4 min/d [95% CI, - 56.0 to -18.8 min/d], respectively) and more physical activity. This study found a large range of individual-level barriers associated with a less active lifestyle in older age, including sociodemographic, behavioral, and health-related factors. These barriers were already evident in midlife, suggesting the importance of early implementation of targeted interventions to promote physical activity and reduce sedentary time.