The apolipoprotein-E4 (APOE*4) and apolipoprotein-E2 (APOE*2) alleles are more common in African American versus non-Hispanic white populations, but relationships of both alleles with Alzheimer’s disease (AD) pathology among African American individuals are unclear. We measured APOE allele and β-amyloid (Aβ) and tau using blood samples and positron emission tomography (PET) images, respectively. Individual regression models tested associations of each APOE allele with Aβ or tau PET overall, stratified by racialized group, and with a racialized group interaction. We included 358 older adults (42% African American) with Aβ PET, 134 (29% African American) of whom had tau PET. APOE*4 was associated with higher Aβ in non-Hispanic white (P < 0.0001), but not African American (P = 0.64) participants; racialized group modified the association between APOE*4 and Aβ (P < 0.0001). There were no other racialized group differences. These results suggest that the association of APOE*4 and Aβ differs between African American and non-Hispanic white populations. Other drivers of AD pathology in African American populations should be identified as potential therapeutic targets.