Most literature linking childhood factors to cognitive health outcomes has focused on educational attainment-defined as years of education attained. However, less has been studied about the other aspects of education, such as early learning problems, and stressful family environments. This study examined whether early learning problems and childhood stressors were associated with mid- and later life cognitive impairment among US adults, and if these associations varied by race. We conducted a cross-sectional analysis using the Health and Retirement Study (HRS) along with respondents' early educational experiences from the 2015 to 2017 Life History Mail Survey (N = 9703). Early learning problems were defined as having any of the following: scholastic problems (reading, writing, mathematics), speaking/language issues, and sensorimotor issues- hearing, vision, speech, and motor-coordination. Cognitive status was classified as three levels (normal, cognitively impaired not demented [CIND], and demented) using the HRS Langa-Weir algorithm. Multinomial logistic regression models using generalized logits, estimated relative risk ratios (RRRs), and 95% confidence intervals (CI) with adjustment for sociodemographic factors. Having at least one early learning problem was associated with increased risk of later life cognitive impairment (RRR: 1.75, 95% CI: 1.34-2.29 for dementia, RRR: 1.42, 95% CI: 1.20-1.67 for CIND). Parental death before the age of 16 was associated with 17% higher risk of CIND in later life (RRR: 1.17, 95% CI: 1.01-1.34). However, learning problem-related differences in risk of cognitive impairment were dependent on race (learning problems × race, p = 0.0001). In the demented group, Blacks were 2.7 times more likely to be demented (RRR: 2.66, 95% CI: 1.69-4.17) amongst older adults that experienced childhood learning problems. Early life exposures predicted risk of cognitive impairment. Policies and interventions that enhance diagnosis of early learning problems and improve childhood social contexts are needed to promote healthy cognitive aging amongst Americans, regardless of race.