Physical activity can improve cognition; however, little is known regarding the relationships between longitudinal objectively-measured physical activity, cognition, and inflammation in older breast cancer survivors. Older (≥60 yrs) breast cancer survivors (n = 216) and frequency-matched non-cancer controls (n = 216) were assessed at baseline (pre-systemic therapy for survivors) and annually for up to five years. Assessments included hip-worn ActiGraphs worn for seven days, neuropsychological tests, the Functional Assessment of Cancer Therapy-Cognitive Function Perceived Cognitive Impairment (FACT-Cog PCI) subscale, and circulating levels of C-reactive protein (CRP) and interleukin-6 (IL-6). Data were analyzed using linear mixed-effect, random-effect contemporaneous fluctuation, and multi-level mediation models, considering covariates; p < .05 (two-sided) was considered significant. Survivors had fewer minutes of moderate-to-vigorous physical activity (MVPA) than controls at 36-, 48-, and 60-month time points (p < .03). Fewer survivors met Aerobic Physical Activity Guidelines at 36 months than controls (17.7% vs 33.0%, p = .030). When Guidelines were met (vs not), FACT-Cog PCI scores were 2.1 ± 1.0 (p = .034) points higher. Higher MVPA and meeting Aerobic Guidelines were not related to objective neuropsychological performance. MVPA was inversely associated with CRP and IL-6 (p < .001), but inflammation did not mediate physical activity effects on perceived cognition. Older breast cancer survivors were less physically active than older non-cancer controls, especially farther from baseline. Meeting Aerobic Guidelines was associated with better perceived cognition in survivors. Survivorship care should consider physical activity monitoring and referral to rehabilitation and supervised exercise programs to promote physical activity and improve recovery in older survivors.