Introduction: Population-based research has consistently shown that people with hearing loss are at greater risk of cognitive impairment. We aimed to explore the cross-sectional association of both subjective and objective hearing measures with global and domain-specific cognitive function. We also examined the influence of hearing aid use on the relationship. Methods: A population-based sample (n = 1,105, 52% women) of 70-year-olds that were representative of the inhabitants of the city of Gothenburg, Sweden completed a detailed cognitive examination, pure-tone audiometry, and a questionnaire regarding perceived hearing problems. A subsample (n = 247, 52% women) also completed a test of speech-recognition-in-noise (SPRIN). Multiple linear regression analyses were conducted to explore the association of hearing with cognitive function, adjusting for sex, education, cardiovascular factors, and tinnitus. Results: Global cognitive function was independently associated with the better ear pure-tone average across 0.5–4 kHz (PTA4, β = −0.13, 95% CI, −0.18, −0.07), the better ear SPRIN score (β = 0.30, 95% CI, 0.19, 0.40), but not with the self-reported hearing measure (β = −0.02, 95% CI, −0.07, 0.03). Both verbally loaded and nonverbally loaded tasks, testing a variety of cognitive domains, contributed to the association. Hearing aid users had better global cognitive function than nonusers with equivalent hearing ability. The difference was only significant in the mild hearing loss category. Discussion: In a population-based sample of 70-year-old persons without dementia, poorer hearing was associated with poorer global and domain-specific cognitive function, but only when hearing function was measured objectively and not when self-reported. The speech-in-noise measure showed the strongest association. This highlights the importance of including standardized hearing tests and controlling for hearing status in epidemiological geriatric research. More research is needed on the role that hearing aid use plays in relation to age-related cognitive declines.