Stressful life events (SLE) have been associated with increased dementia risk, but their association with cognitive decline has been inconsistent. In a longitudinal population-based study of older individuals, we examined the association between SLE and cognitive decline, and the role of potential effect modifiers. A total of 2665 non-demented participants of the Cache County Memory Study completed an SLE questionnaire at Wave 2 and were revisited 4 and 7 years later. The events were represented via several scores: total number, subjective rating (negative, positive, and unexpected), and a weighted summary based on their impact. Cognition was assessed at each visit with the modified Mini-Mental State Exam. General linear models were used to examine the association between SLE scores and cognition. Effect modification by age, education, and APOE genotype was tested. Years of formal education (p = 0.006) modified the effect of number of SLE, and age (p = 0.009) modified the effect of negative SLE on the rate of cognitive decline. Faster decline was observed among those with fewer years of education experiencing more SLE and also among younger participants experiencing more negative SLE. There was no association between other indicators of SLE and cognitive decline. APOE genotype did not modify any of the aforementioned associations. The effects of SLE on cognition in late life are complex and vary by individual factors such as age and education. These results may explain some of the contradictory findings in the literature.