Alzheimer's disease (AD) and cerebral small vessel disease (cSVD), the two most common causes of dementia, are characterized by white matter (WM) alterations diverging from the physiological changes occurring in healthy aging. Diffusion tensor imaging (DTI) is a valuable tool to quantify WM integrity non-invasively and identify the determinants of such alterations. Here, we investigated main effects and interactions of AD pathology, APOE-ε4, cSVD, and cardiovascular risk on spatial patterns of WM alterations in non-demented older adults. Within the prospective European Prevention of Alzheimer's Dementia study, we selected 606 participants (64.9 ± 7.2 years, 376 females) with baseline cerebrospinal fluid samples of amyloid β1-42 and p-Tau181 and MRI scans, including DTI scans. Longitudinal scans (mean follow-up time = 1.3 ± 0.5 years) were obtained in a subset (n = 223). WM integrity was assessed by extracting fractional anisotropy and mean diffusivity in relevant tracts. To identify the determinants of WM disruption, we performed a multimodel inference to identify the best linear mixed-effects model for each tract. AD pathology, APOE-ε4, cSVD burden, and cardiovascular risk were all associated with WM integrity within several tracts. While limbic tracts were mainly impacted by AD pathology and APOE-ε4, commissural, associative, and projection tract integrity was more related to cSVD burden and cardiovascular risk. AD pathology and cSVD did not show any significant interaction effect. Our results suggest that AD pathology and cSVD exert independent and spatially different effects on WM microstructure, supporting the role of DTI in disease monitoring and suggesting independent targets for preventive medicine approaches.