Cognitive deficits are commonly reported after COVID-19 recovery, but little is known in the older population. This study aims to investigate possible cognitive damage in older adults 6 months after contracting COVID-19, as well as individual risk factors. This cross-sectional study involved 70 participants aged 60-78 with COVID-19 6 months prior and 153 healthy controls. Montreal Cognitive Assessment-Basic (MoCA-B) screened for cognitive impairment; Geriatric Depression Scale and Geriatric Anxiety Inventory screened for depression and anxiety. Data were collected on demographics and self-reports of comorbid conditions. The mean age of participants was 66.97 ± 4.64 years. A higher proportion of individuals in the COVID group complained about cognitive deficits (χ2 = 3.574; p = 0.029) and presented with deficient MoCA-B scores (χ2 = 6.098, p = 0.014) compared to controls. After controlling for multiple variables, all the following factors resulted in greater odds of a deficient MoCA-B: COVID-19 6-months prior (OR, 2.44; p = 0.018), age (OR, 1.15; p < 0.001), lower income (OR, 0.36; p = 0.070), and overweight (OR, 2.83; p = 0.013). Further analysis pointed to individual characteristics in COVID-19-affected patients that could explain the severity of the cognitive decline: age (p = 0.015), lower income (p < 0.001), anxiety (p = 0.049), ageusia (p = 0.054), overweight (p < 0.001), and absence of cognitively stimulating activities (p = 0.062). Our study highlights a profile of cognitive risk aggravation over aging after COVID-19 infection, which is likely mitigated by wealth but worsened in the presence of overweight. Ageusia at the time of acute COVID-19, anxiety, being overweight, and absence of routine intellectual activities are risk factors for more prominent cognitive decline among those infected by COVID-19.