Genome-wide association studies have discovered common genetic variants associated with cognitive performance. Polygenic scores that summarize these discoveries explain up to 10% of the variance in cognitive test performance in samples of adults. However, the role these genetics play in cognitive aging is not well understood. We analyzed data from 168 cognitively healthy participants aged 23-77 years old, with data on genetics, neuropsychological assessment, and brain-imaging measurements from two large ongoing studies, the Reference Abilities Neural Networks, and the Cognitive Reserve study. We tested whether a polygenic index previously related to cognition (Cog PGI) would moderate the relationship between age and measurements of the cognitive domains extracted from a neuropsychological evaluation: fluid reasoning, memory, vocabulary, and speed of processing. We further explored the relationship of Cog PGI and age on cognition using Johnson-Neyman intervals for two-way interactions. Sex, education, and brain measures of cortical thickness, total gray matter volume, and white matter hyperintensity were considered covariates. The analysis controlled for population structure-ancestry. There was a significant interaction effect of Cog PGI on the association between age and the domains of memory (Standardized coefficient = -0.158, p-value = 0.022), fluid reasoning (Standardized coefficient = -0.146, p-value = 0.020), and vocabulary (Standardized coefficient = -0.191, p-value = 0.001). Higher PGI strengthened the negative relationship between age and the domains of memory and fluid reasoning while PGI weakened the positive relationship between age and vocabulary. Based on the Johnson-Neyman intervals, Cog PGI was significantly associated with domains of memory, reasoning, and vocabulary for younger adults. There is a significant moderation effect of genetic predisposition for cognition for the association between age and cognitive performance. Genetics discovered in genome-wide association studies of cognitive performance show a stronger association in young and midlife older adults.