Heterogeneity in the progression of cognitive impairment, which is common in sporadic Alzheimer's disease (AD) trials, is especially challenging to predict in pre-symptomatic populations, and has a negative impact on clinical trial power. Heritability of AD extends beyond the APOE genotype, with multiple common genetic variants identified in large genome wide association studies (GWAS). Incorporating APOE and additional genetic variants into models has the potential to improve the prediction of disease progression. The limited volume of genetic cohorts in AD and the sample overlap in GWAS summary statistics poses some concern for polygenic risk score (PRS) overfitting. A PRS was calculated using genome-wide association study (GWAS) summary statistics for clinical AD diagnosis. PRS derived from this GWAS study was computed for participants drawn from two aging studies. Logistic regression models assessed the association between PRS and Mini Mental State Exam (MMSE) decline covarying for age, sex, education, APOE-ε4, and baseline MMSE score. Age and sex interactions with PRS were also assessed. Samples assumed to be used in the GWAS calculations were then removed from the test set and the impact to performance was analyzed. Participants in the training and test set showed similar baseline ages, years of education and baseline MMSE scores in the whole sample and after stratification by APOE4 carrier status. The PRS model was a significant predictor of MMSE decline, as well as in APOE4 carriers and noncarriers. Model performance was compared to the test set excluding the GWAS overlap samples and no significant difference was observed. The proposed model including PRS explains heterogeneity in cognitive decline above and beyond the APOE4 allele. Testing the model in a dataset excluding GWAS overlap samples did not result in a significant difference in performance and potential overfitting does not appear to be an impact. Utilization of demographic and genomic factors beyond APOE in PRS models could enhance clinical trial recruitment and stratification strategies for trial analyses, such that APOE4 carriers are selected for probable cognitive decline, in addition to APOE3 carriers that are high on polygenic risk.