Higher midlife physical activity engagement has been associated with lower dementia risk in late life. However, the underlying mechanisms contributing to the protective effect remain unclear. The goal of the current study was to evaluate the associations of physical activity with cerebral amyloid-β (Aβ) and tau in a predominately middle-aged community-based cohort, as well as to explore whether the associations differ by sex or age. Participants from the Framingham Heart Study underwent 11C-Pittsburgh Compound B amyloid and 18F-Flortaucipir tau positron emission tomography (PET) imaging. Total physical activity levels were evaluated by self-report using the Physical Activity Index (PAI). Cross-sectional associations between total PAI with regional Aβ and tau PET retention were evaluated using linear regression models adjusted for demographic and cardiovascular risk factors. Interactions with sex and age group were examined and stratified analyses were performed when significant. FDR-correction for multiple comparisons was applied. The sample included 354 participants (mean age 53±8 years, 51% female). Higher total PAI scores were associated with lower entorhinal cortex tau PET binding (β(SE) = -0.021(0.008), p = 0.049). There were significant interactions with sex. In men alone, total PAI inversely associated with entorhinal cortex (β(SE) = -0.035(0.009), p = 0.001), inferior temporal (β(SE) = -0.029(0.010), p = 0.012), and rhinal cortex tau(β(SE) = -0.033(0.010), p = 0.002). The results suggest that higher midlife physical activity engagement may confer resistance to tau pathology. However, the effects may vary based on sex, highlighting the importance of better understanding and tailoring lifestyle interventions to address sex disparities.