Introduction: As the human lifespan rises, older people are increasingly affected by multimorbidity, including hearing impairment (HI), frailty, and cognitive decline. Objective: We studied the effect of HI on the cognitive performance of older patients with mild cognitive impairment and mild dementia and how this effect is altered when HI and frailty coexist. Methods: The sample comprised 172 outpatients (age ≥65 years) with a Clinical Dementia Rating of 0.5–1 and a Mini-Mental State Examination score ≥20. Cognitive, neuropsychological, physical function tests, and body measurements were performed. The participants’ comorbidity indices were calculated, and they were asked to report their hearing condition. Finally, the participants were divided into 4 groups based on the presence of frailty, HI, both conditions, or neither. Results: The presence of HI was significantly associated with a lower Digit Symbol Substitution Test (DSST) score (unadjusted model: p = 0.003; adjusted: p = 0.012). Moreover, people with both HI and frailty were more likely to have a low DSST score than those in the other groups (unadjusted model: OR: 3.741, 95% CI: 1.374–10.183, p = 0.010; adjusted model: OR: 4.192, 95% CI: 1.026–17.136, p = 0.046). Conclusions: This study highlights the negative relationship between cognitive performance scored on the DSST and the presence of HI. Furthermore, the presence of both frailty and HI is associated with a worse cognitive score than HI alone.