Abstract Later-life cognitive function is influenced by genetics as well as early- and later-life socioeconomic context. However, few studies have examined the interaction between genetics and early childhood factors. Using gene-based tests (interaction sequence kernel association test [iSKAT]/iSKAT optimal unified test), we examined whether common and rare variants in 39 gene regions previously associated with cognitive performance, dementia, and related traits had an interaction with childhood socioeconomic context (parental education and financial strain) on memory performance or decline in European ancestry (EA, N = 10,468) and African ancestry (AA, N = 2,252) participants from the Health and Retirement Study. Of the 39 genes, 22 in EA and 19 in AA had nominally significant interactions with at least one childhood socioeconomic measure on memory performance and/or decline; however, only one, SLC24A4 x fatherâs education, remained significant after multiple testing correction (false discovery rate [FDR] < .05). In trans-ethnic meta-analysis, 2 genes interacted with childhood socioeconomic context: MS4A4A x motherâs education on memory performance, and SLC24A4 x fatherâs education on memory decline. Both interactions remained significant after adjusting for respondentâs own education, apolipoprotein-Δ4 allele status, lifestyle factors, body mass index, and comorbidities. For both interactions in EA and AA, the genetic effect was stronger in participants with low parental education. Examination of common and rare variants in genes discovered through genome-wide association studies shows that childhood context may interact with gene regions to jointly impact later-life memory function and decline. Genetic effects may be more salient for those with lower childhood socioeconomic status.