Colorectal polyps are the precursor lesions of most colorectal cancers. This study aimed to evaluate associations between bone mineral density (BMD), metabolic syndrome (MetS), and gastrointestinal diseases with colorectal polyps in middle-aged and elderly populations. A retrospective cross-sectional study was performed using data from the health examination database of a tertiary medical center in southern Taiwan in 2015. Subjects aged 50 years and older who had been assessed for metabolic factors and had undergone colonoscopy, upper gastrointestinal endoscopy, and dual energy X-ray absorptiometry scan (DEXA) were included. Factors associated with colorectal polyps were evaluated using univariate and multivariate logistic regression. In total, 1515 subjects were included, with mean age 60.1 years. Among them, 710 (46.9%) had colorectal polyps. Multivariate logistic regression analysis showed that high fasting glucose (OR = 1.08, p = 0.001), high triglycerides (OR = 1.02, p = 0.008), high total cholesterol (OR = 1.004, p = 0.009), reflux esophagitis (OR = 1.44, p = 0.002), duodenal polyps (OR = 1.75, p = 0.026), gastric ulcer (OR = 1.38, p = 0.024), duodenal ulcers (OR = 1.45, p = 0.028), osteopenia (OR = 1.48, p = 0.001), and MetS (OR = 1.46, p < 0.001) were independently associated with colorectal polyps. In conclusion, hyperglycemia, hypercholesterolemia, hypertriglyceridemia, MetS, duodenal polyps, gastric and duodenal ulcers, reflux esophagitis, and low BMD are independent risk factors associated with colorectal polyps in the middle-aged and elderly Taiwanese population.