BackgroundAssociations of Apolipoprotein (APOE) ε2 or ε4 (APOE2 or APOE4) dosages with cognitive change may differ across racial groups.MethodsLongitudinal data on 1770 middle-aged White and African American adults was compiled from the Healthy Aging in Neighborhoods of Diversity across the Life Span (HANDLS 2004-2013) study. APOE2 and APOE4 dosages were the two main exposures, while v1 and annual rate of change in cognitive performance (between v1 and v2) on 11 test scores were the main outcomes of interest (v1: 2004–2009 and v2: 2009–2013). Mixed-effects linear regression models were conducted adjusting for socio-demographic, lifestyle, and health-related potential confounders. Race (African American vs. White) and sex within racial groups were main effect modifiers.ResultsUpon adjustment for multiple testing and potential confounders, APOE4 allelic dosage was associated with faster decline on a test of verbal memory among Whites only (CVLT-List A: γ12 = − 0.363 ± 0.137, p = 0.008), but not among African Americans. In contrast, among African American women, APOE4 dosage was linked to slower decline on a test of attention (BTA: γ12 = + 0.106 ± 0.035, p = 0.002), while no association was detected among African American men. APOE2 and APOE4 dosages showed inconsistent results in other domains of cognition overall and across racial groups that did not survive correction for multiple testing.ConclusionsIn conclusion, APOE4 dosage was associated with faster decline on a test of verbal memory among Whites only, while exhibiting a potential protective effect among African American women in the domain of attention. Further longitudinal studies are needed to replicate our race and sex-specific findings.