This study tests whether memory impairment and perceived increases in stress due to life changes independently contribute to similar personality changes, such as increased neuroticism and decreased conscientiousness. Longitudinal data from 12,912 participants aged 50+ from the Health and Retirement Study (2006–2020) were analyzed using Latent Growth Curve Models (LGCMs). Six LGCMs were simultaneously estimated to examine how changes in personality across three data points spanning 8 years are predicted by both the classification of memory impairment (MI) statuses assessed via the Telephone Interview for Cognitive Status and estimated increases in life stress. Decreases were observed in all Big Five personality factors (B = − 0.240/− 0.510, SE = 0.085/0.089) over the 8-year period. No significant differences in personality changes were found between MI cases and non-MI cases. Increased life stress was significantly associated with a rise in neuroticism (B = 0.587, SE = 0.094) and reductions in conscientiousness, extraversion, agreeableness, and openness (B ranges from − 0.223 to − 0.497). The findings emphasize adaptive/developmental shifts in personality in response to life stress rather than neuropathological ones related to memory impairment. Clinical assessment of personality changes in mid-to-late life should first consider psychological maladaptation to situational threats and challenges.