Alzheimer’s disease (AD) risk is increased in carriers of the apolipoprotein E (APOE) ε4 allele and decreased in ε2 allele carriers compared with the ε3ε3 genotype. The aim of this study was to determine whether: the APOE genotype affects brain grey (GM) or white matter (WM) structure; and if differences exist, the age when they become apparent and whether there are differential effects by sex. We used cross-sectional magnetic resonance imaging data from ~43,000 (28,494 after pre-processing) white British cognitively healthy participants (7,446 APOE ε4 carriers) aged 45–80 years from the UK Biobank cohort and investigated image-derived phenotypes (IDPs). We observed no statistically significant effects of APOE genotype on GM structure volumes or median T2* in subcortical structures, a measure related to iron content. The volume of white matter hyperintensities differed significantly between APOE genotype groups with higher volumes in APOE ε4ε4 (effect size 0.14 standard deviations [SD]) and ε3ε4 carriers (effect size 0.04 SD) but no differences in ε2 carriers compared with ε3ε3 carriers. WM integrity measures in the dorsal (mean diffusivity [MD]) and ventral cingulum (MD and intracellular volume fraction), posterior thalamic radiation (MD and isotropic volume fraction) and sagittal stratum (MD) indicated lower integrity in APOE ε4ε4 carriers (effect sizes around 0.2–0.3 SD) and ε3ε4 (effect sizes around 0.05 SD) carriers but no differences in ε2 carriers compared with the APOE ε3ε3 genotype. Effects did not differ between men and women. APOE ε4 homozygotes had lower WM integrity specifically at older ages with a steeper decline of WM integrity from the age of 60 that corresponds to around 5 years greater “brain age”. APOE genotype affects various white matters measures, which might be indicative of preclinical AD processes. This hypothesis can be assessed in future when clinical outcomes become available.