Loss of white matter integrity (WMI) is associated with gait deficits in middle-aged and older adults. However, these deficits are often only apparent under cognitively demanding situations, such as walking and simultaneously performing a secondary cognitive task. Moreover, evidence suggests that declining executive functions (EF) are linked to gait decline, and their co-occurrence may point to a common underlying pathology, i.e., degeneration of shared brain regions. In this study, we applied diffusion tensor imaging (DTI) and a standardized gait assessment under single- and dual-tasking (DT) conditions (walking and subtracting) in 74 middle-aged and older adults without any significant gait or cognitive impairments to detect subtle WM alterations associated with gait decline under DT conditions. Additionally, the Trail Making Test (TMT) was used to assess EF, classify participants into three groups based on their performance, and examine a possible interaction between gait, EF, and WMI. Gait speed and subtracting speed while dual-tasking correlated significantly with the fractional anisotropy (FA) in the bilateral anterior corona radiata (highest r = 0.51/p < 0.0125 FWE-corrected). Dual-task costs (DTC) of gait speed correlated significantly with FA in widespread pathways, including the corpus callosum, bilateral anterior and superior corona radiata, as well as the left superior longitudinal fasciculus (highest r = −0.47/p < 0.0125 FWE-corrected). EF performance was associated with FA in the left anterior corona radiata (p < 0.05); however, EF did not significantly mediate the effects of WMI on DTC of gait speed. There were no significant correlations between TMT and DTC of gait and subtracting speed, respectively. Our findings indicate that gait decline under DT conditions is associated with widespread WM deterioration even in middle-aged and older adults without any significant gait or cognitive impairments. However, this relationship was not mediated by EF.