Semantic cognition is thought to involve the interaction of heteromodal conceptual representations with control processes that (i) focus retrieval on currently-relevant information, and (ii) suppress dominant yet irrelevant features and associations. Research suggests that semantic control demands are higher when retrieving a link between weakly-associated word pairs, since there is a mismatch between the pattern of semantic retrieval required by the task and the dominant associations of individual words. In addition, given that heteromodal concepts are thought to reflect the integration of vision, audition, valence and other features, the control demands of semantic tasks should be higher when there is less consistency between these features. In the present study, 62 volunteers completed a semantic decision task, where association strength and semantic-affective congruence were manipulated. We used diffusion tensor magnetic resonance imaging to obtain fractional anisotropy (FA) measures of white matter tracts hypothesized to be part of the semantic network. The behavioural data revealed an interaction between semantic-affective congruence and strength of association, suggesting these manipulations both contribute to semantic control demands. Next we considered how individual differences in these markers of semantic control relate to the microstructure of canonical white matter tracts, complementing previous studies that have largely focused on measures of intrinsic functional connectivity. Repeated-measures analysis of covariance showed opposing interactions between semantic control markers and FA of two tracts: left inferior longitudinal fasciculus (ILF) and right inferior fronto-occipital fasciculus (IFOF). Participants with higher FA in left ILF showed more efficient retrieval of weak associations, and more accurate performance for weak associations when meaning and valence were incongruent, consistent with the hypothesis that this left hemisphere tract supports semantic control. In contrast, participants with higher FA in right IFOF were more accurate for trials in which meaning and valence were congruent, and consequently when semantic control demands were minimised. These findings are consistent with recent studies showing that semantic control processes are strongly left-lateralised. In contrast, long-range connections from vision to semantic regions in the right hemisphere might support relatively automatic patterns of semantic retrieval.