Age-related cognitive decline has become an increasingly relevant public health issue. However, risk and protective factors of cognitive decline have yet to be investigated prospectively taking into account genetic, lifestyle, physical and mental health factors. Population-based data from middle-aged (40 to 59 years; N = 2,764) and older individuals (60 to 80 years; N = 1,254) were drawn from a prospective community cohort study using the Tower of London (TOL) planning task. Assessments were repeated at a 5-year interval to investigate age-related changes in planning performance and to determine the impact of risk and protective factors. Planning performance improved in middle-aged, but declined in older participants over 5 years. SNPs affecting the dopamine system (COMT, DRD2) and APOE polymorphisms differentially predicted cognitive performance in older vs. middle-aged individuals. For older individuals, high alcohol consumption, antidepressant medication and living without a partner had additional negative predictive power on cognition. In contrast, undiagnosed hypertension, no obstructive lung disease, and fewer years of education predicted cognitive decline in the middle-aged group. The results inform screening for individuals particularly vulnerable to cognitive decline and interventions (e.g., focusing on lifestyle factors) to help maintain cognitive performance into old age.