Abstract Background and Objectives Alzheimer’s disease and related dementias (ADRD) remain a pressing concern in the US, which also has one of the highest incarceration rates worldwide. Existing research has analyzed dementia risk and care among currently incarcerated and homeless populations; this paper fills a gap by examining later-life cognitive disparities facing formerly incarcerated and/or homeless individuals. Research Design and Methods Using Health and Retirement Study data (HRS; 1998–2018), we tested whether formerly homeless and/or incarcerated people had earlier onset of cognitive decline, and whether they were more likely to have modifiable risk factors for ADRD than those without such experiences. We estimated prevalence of mild cognitive impairment (MCI), dementia, and self-reported ADRD diagnosis. We also used repeated observations of cognitive functioning coupled with nested nonlinear regression to examine the onset of accelerated cognitive decline. Results Adjusting for demographic variables, formerly homeless and/or incarcerated participants have increased odds of having risk factors for dementia, including smoking, drinking alcohol, depression, mental illness, and discrimination from healthcare providers and police. Diagnosis of MCI and ADRD were more common among participants reporting a history of incarceration or homelessness. Nested nonlinear regression revealed that previous experiences of incarceration or homelessness—alone or together—are associated with earlier onset of accelerated cognitive decline compared to those who experienced neither. Discussion and Implications This study suggests that, even if they occur earlier in life, homelessness and incarceration experiences may increase risk of poorer cognitive health in late life. Future work is warranted to understand the lifelong consequences linked to this adversity.