While obesity, rapid weight loss, and sedentary lifestyles contribute to gallstones, the impacts of long-term weight patterns and physical activity on gallstone risk have not been extensively studied at the population level. Using Korea’s population-based health database, we created a cohort of 5,062,154 subjects who received over five consecutive biannual health check-ups from 2002 to 2018. Gallstone risk was calculated using hazard ratios (HRs) based on weight patterns (gain, loss, and cycling) and physical activity. Adjustments were made for covariates, such as body mass index (BMI), waist circumference, diabetes, and life style factors. Subgroup analysis was performed by sex and age groups (< 45 and ≥ 45 years). All groups with weight changes showed increased HRs 1.32 (95% CI 1.07–1.65) for > 20% weight loss, 1.13 (0.96–1.33) for > 20% weight gain, 1.04 (1.02–1.06) for weight cycling, 1.03 (1.01–1.05) for 5–20% weight gain, and 1.02 (1.00–1.04) for 5–20% weight loss. Risk increased with weight loss or gain in underweight or overweight BMI categories, respectively, and in any weight change for normal BMI. Males and older individuals had higher risks with weight loss, while females and younger individuals had higher risks with weight gain. Weight cycling increased risk for both sexes. Regular physical activity reduced gallstone risk in weight changers to levels similar to weight maintainers, with the highest reduction in those with > 20% weight gain (61%) and the lowest in weight cycling (1%). The risks of gallstones associated with weight gain, loss, or cycling can be mitigated by regular physical activity. The risks vary by sex and age.