Current evidence showed that heavy metal exposure including cadmium (Cd) exposure might contribute to memory function impairment in youth, while this association has not been extensively explored in senior groups. Complementary therapy like physical activity (PA) is proved to enhance memory; however, the combined effects of Cd exposure and PA are interesting issues worth investigating. Cross-sectional data from the National Health and Nutrition Examination Survey (NHANES) 2011-2014 were analyzed. Multivariable weighted linear regression model and restricted cubic splines analysis were used to examine the association between blood Cd, PA, and memory function. Ultimately, 1884 samples were analyzed, and the weighted participants were 98,350,183. Results showed that in the immediate and delayed recall tests, a negative association was found between blood Cd and scores for the fully adjusted model, while a positive association was detected with PA on memory test scores. In subgroup analysis, for the delayed recall test, in lower Cd exposure (Cd = Q1), stronger effect size was found in the moderate level PA group than the higher level PA group (moderate level PA group, β = 1.133, 95% CI: 0.330, 1.936; high level PA group, β = 0.203, 95% CI: - 0.314, 0.719), and this finding also existed in higher (Cd = Q4) exposure (moderate level PA group, β = 0.988, 95% CI: 0.267, 1.708; high level PA group, β = 0.830, 95% CI: 0.261, 1.400). Moreover, the non-linear relationship between Cd exposure and performance of CERAD test under different levels of PA was reported, and the moderate level PA group performed best from lower to higher blood Cd. According to our research, the benefit of PA did not always expand with the PA intensity increment under different Cd exposure. Performing an appropriate level of physical exercise could alleviate the memory decline under Cd exposure in the elder groups. Further biological investigations are warranted to verify these findings.