The Long Life Family Study (LLFS) is a family based, prospective study of healthy aging and familial longevity. The study includes two assessments of cognitive function that were administered approximately 8 years apart. To test whether APOE genotype is associated with change of cognitive function in older adults. We used Bayesian hierarchical models to test the association between APOE alleles and change of cognitive function. Six longitudinally collected neuropsychological test scores were modelled as a function of age at enrollment, follow-up time, gender, education, field center, birth cohort indicator (≤1935, or >1935), and the number of copies of ɛ2 or ɛ4 alleles. Out of 4,587 eligible participants, 2,064 were male (45.0%), and age at enrollment ranged from 25 to 110 years, with mean of 70.85 years (SD: 15.75). We detected a significant cross-sectional effect of the APOEɛ4 allele on Logical Memory. Participants carrying at least one copy of the ɛ4 allele had lower scores in both immediate (-0.31 points, 95% CI: -0.57, -0.05) and delayed (-0.37 points, 95% CI: -0.64, -0.10) recall comparing to non-ɛ4 allele carriers. We did not detect any significant longitudinal effect of the ɛ4 allele. There was no cross-sectional or longitudinal effect of the ɛ2 allele. The APOEɛ4 allele was identified as a risk factor for poorer episodic memory in older adults, while the APOEɛ2 allele was not significantly associated with any of the cognitive test scores.