Abstract Background Many studies have suggested that cholelithiasis or cholecystectomy may be associated with risk of colorectal cancer [CRC]. This study aimed to verify CRC risk among cholelithiasis patients who experienced cholecystitis or cholecystectomy or both. Methods Data of a cholelithiasis cohort (n = 28,267) and a noncholelithiasis cohort (n = 112,948) were retrieved (1:4 ratio) from the Longitudinal Health Insurance Database for comparison of CRC risk. Confounding factors were adjusted in multivariate analysis. A hospital-based cancer registry containing 8156 CRC patients was also examined. Data were statistically analyzed using χ2 test, t test, and Cox proportional-hazards model. Results Univariate and multivariate analysis showed that having cholelithiasis was associated with increased risk of CRC (crude hazard ratio: 1.533 [95% confidence interval {CI}: 1.423–1.652); adjusted HR: 1.481 [95% CI: 1.375–1.597], respectively, p < 0.001) relative to not having cholelithiasis. Increased CRC incidence among patients with cholelithiasis was higher in the right-sided colon (adjusted HR: 1.452 [95% CI: 1.234–1.709], p < 0.001), and higher in female patients. The trend of higher incidence of right-sided colon cancer was also seen in the hospital-based cancer registry. Of note, cholecystitis or cholecystectomy was associated with mildly increased CRC risk among cholelithiasis patients. Cholecystitis was associated with increased risk of left-sided CRC. Cholecystitis plus cholecystectomy was associated with increased risk of rectal cancer. Conclusions These results support that cholelithiasis is associated with increased risk of CRC especially in the right-sided colon and among female patients. Cholecystitis and cholecystectomy may shift cancer to the distal part of the large bowel.