Dietary fiber is a possible nutritional component which aids in the prevention of visceral fat accumulation. We examined the association between dietary fiber intake and visceral fat volume (VFV) by sex, and further analysed the association by major food sources of dietary fiber. In this cross-sectional study, we measured VFV in 2779 Japanese (1564 men and 1215 women) aged 40-89 who underwent positron emission tomography/computed tomography for cancer screening between 2004 and 2005. Dietary fiber intake was calculated based on a validated semi-quantitative food frequency questionnaire. The association between dietary fiber intake and VFV was investigated using multivariate linear regression models after adjustment for potential confounders. Total, soluble, and insoluble fiber intakes were inversely associated with VFV in men (Q1: 3740 cm3, Q4: 3517 cm3, Ptrend: 0.0006 for total fiber), but not in women (Q1: 2207 cm3, Q4: 2193 cm3,Ptrend: 0.88 for total fiber). Statistically significant sex difference was observed (Pinteraction = 0.001 for total fiber). Subgroup analyses by major food sources revealed that dietary fiber intakes from beans, vegetables and fruits showed an inverse association with VFV in men, while cereal fiber intake showed a tendency toward a positive association in both sexes (Q1: 3520 cm3, Q4: 3671 cm3, Ptrend: 0.05 in men, Q1: 2147 cm3, Q4: 2227 cm3, Ptrend: 0.10 in women). We observed a sex-specific association between dietary fiber intake and VFV in Japanese adults. This study suggests that efforts against visceral fat accumulation should take account of the source of dietary fiber.