To explore the relationship between physical activity (PA) and quality of life (QOL) among Hispanic and non-Hispanic white breast cancer (BC) cases and population-based controls from the New Mexico 'Long-Term Quality of Life Study'. Self-reported PA (low, moderate, vigorous MET hours/week) at baseline and follow-up interviews (12-15years) were available for 391 cases and controls and modeled using multiple linear regressions with SF-36 mean composite scores for physical and mental health. The change in PA from baseline to follow-up and interactions with ethnicity were also examined. Models were adjusted for age at diagnosis/baseline interview, education, comorbidities, body mass index, and change in PA. PA intensities at each timepoint did not differ by case/control status; however, the change in vigorous PA was lower among cases (p = 0.03). At follow-up, low intensity PA increased mental health QOL scores among cases; however, the interaction between low intensity PA and ethnicity was statistically significant among controls indicating decreased mental health among Hispanics (p = 0.02). Change in moderate PA was associated with increased physical and mental health among cases (physical: β = 0.186, p = 0.008; mental: β = 0.225, p = 0.001) and controls (physical: β = 0.220, p < 0.0001; mental: β = 0.193, p = 0.002), when controlling for confounders. Our results demonstrate that all levels of PA are important for mental health among BC cases, while activities of higher intensity are important for physical health among women overall. The statistical interaction observed between ethnicity and low intensity PA among controls for mental health warrants further research to provide a meaningful interpretation.