Alzheimer’s disease (AD) is the most common form of dementia and the leading risk factor, after age, is possession of the apolipoprotein E epsilon 4 allele (APOE4). Approximately 50% of AD patients carry one or two copies of APOE4 but the mechanisms by which it confers risk are still unknown. APOE4 carriers are reported to demonstrate changes in brain structure, cognition, and neuropathology, but findings have been inconsistent across studies. In the present study, we used multi-modal data to characterise the effects of APOE4 on the brain, to investigate whether AD pathology manifests differently in APOE4 carriers, and to determine if AD pathomechanisms are different between carriers and non-carriers. Brain structural differences in APOE4 carriers were characterised by applying machine learning to over 2000 brain MRI measurements from 33,384 non-demented UK biobank study participants. APOE4 carriers showed brain changes consistent with vascular dysfunction, such as reduced white matter integrity in posterior brain regions. The relationship between APOE4 and AD pathology was explored among the 1260 individuals from the Religious Orders Study and Memory and Aging Project (ROSMAP). APOE4 status had a greater effect on amyloid than tau load, particularly amyloid in the posterior cortical regions. APOE status was also highly correlated with cerebral amyloid angiopathy (CAA). Bulk tissue brain transcriptomic data from ROSMAP and a similar dataset from the Mount Sinai Brain Bank showed that differentially expressed genes between the dementia and non-dementia groups were enriched for vascular-related processes (e.g., “angiogenesis”) in APOE4 carriers only. Immune-related transcripts were more strongly correlated with AD pathology in APOE4 carriers with some transcripts such as TREM2 and positively correlated with pathology severity in APOE4 carriers, but negatively in non-carriers. Overall, cumulative evidence from the largest neuroimaging, pathology, and transcriptomic studies available suggests that vascular dysfunction is key to the development of AD in APOE4 carriers. However, further studies are required to tease out non-APOE4-specific mechanisms.