Background: An association between nonalcoholic fatty liver disease (NAFLD) and low vitamin D levels has been suggested. We investigated the relationship between vitamin D and NAFLD assessed by controlled attenuation parameter (CAP). Methods: We conducted a retrospective cohort study of apparently healthy subjects who underwent Fibroscan during health screening tests. NAFLD was diagnosed using CAP values. Results: Among the 1202 subjects (mean age 57.2 years, 60.6% male), 630 (52.4%) subjects had NAFLD with CAP ≥ 248 dB/m. Multivariable analysis was conducted after adjusting for metabolic risk factors including diabetes, hypertension, hypercholesterolemia, body mass index, high-density lipoprotein cholesterol, triglyceride and smoking. Higher vitamin D levels showed a lower risk of NAFLD compared to the lowest quartile of vitamin D in a dose-dependent manner (OR 0.68, 95% CI 0.47–1.00 in Q2 vs. Q1; OR 0.65, 95% CI 0.44–0.94 in Q3 vs. Q1; and OR 0.64, 95% CI 0.44–0.94 in Q4 vs. Q1). The highest quartile of vitamin D showed a decreased risk of a severe grade of steatosis (CAP ≥ 302 dB/m) compared to the lowest quartile (OR 0.52, 95% CI 0.31–0.87 in Q4 vs. Q1). Conclusions: Higher levels of serum vitamin D were associated with a decreased risk of CAP-defined NAFLD, compared to low levels of serum vitamin D. The association between NAFLD and vitamin D suggests that vitamin D may exert a protective role against NAFLD.