Background: Physical activity has been associated with a lower risk of various types of cancer and reduced cancer-specific mortality. Less is known about its impact on pancreatic cancer. The aim of this systematic review and meta-analysis was to summarize evidence on the association between physical activity and pancreatic cancer risk and mortality. Methods: PubMed and Embase were searched until May 2024 for studies examining physical activity in relation to pancreatic cancer incidence and mortality. Summary risk estimates for highest vs. lowest physical activity levels were calculated using a random-effects model. The risk of publication bias was assessed with a funnel plot and Egger’s regression test. Results: A total of seven case–control and eighteen prospective cohort studies were included that investigated the association between physical activity and pancreatic cancer incidence. Our meta-analysis showed a summary estimate of 0.75 (95% CI 0.64–0.88) for case–control studies (I2 = 23%, n = 7) and a summary estimate of 0.91 (95% CI 0.86–0.97) for prospective cohort studies (I2 = 5%, n = 18). Among the six prospective cohort studies that assessed pancreatic cancer mortality, the summary estimate was 1.03 (95% CI 0.83–1.27), I2 = 50%. Conclusions: Higher levels of physical activity were associated with reduced pancreatic cancer risk. Evidence from a limited number of studies suggests that pre-diagnosis physical activity does not affect pancreatic cancer mortality.