Evidence from functional imaging studies points to a role for gender in language ability. However, recent studies suggest that sex differences in the neural basis of language are still unclear, reflecting a complex interaction between sex and language ability. We used diffusion weighted magnetic resonance imaging and global probabilistic tractography to investigate white matter (WM) pathways between 32 male and 35 age- and IQ-matched female adult participants in relation to their verbal abilities. Males showed higher fractional anisotropy (FA) in the left anterior thalamic radiations (ATR), right cingulum-angular bundle, right corticospinal tract, bilateral superior longitudinal fasciculus-temporal terminations, bilateral uncinate fasciculus (UNC), and corpus callosum-forceps minor when compared with the female group. In contrast, females showed higher radial diffusivity (RD) in the left ATR and left UNC when compared to the male group. The relationship between WM metrics and verbal ability also differed across the two groups: a negative correlation between verbal comprehension index (VCI) and FA as well as axial diffusivity (AD) in left cingulum-cingulate gyrus (CCG) supracallosal bundle in males but not in females; a negative correlation between verbal IQ (VIQ) and FA in the right corticospinal tract (CST), and a positive correlation between VCI and RD in corpus callosum-forceps minor in the female but not in the male group. A direct comparison of these correlation coefficients yielded significant differences between the groups for the VCI-AD and VIQ -FA associations. The findings may reflect sex differences in WM related to language ability.