Decreased smell could cause appetite suppression and malnutrition. However, there is a paucity of longitudinal data between olfaction and nutritional status in older adults. We aimed to prospectively examine the relationship between olfactory impairment and overall diet quality (reflecting adherence to dietary guidelines) in a population-based cohort of older adults. We used 5-year follow-up data from 557 adults (aged 60+ years at baseline) whose olfaction was measured using the San Diego Odor Identification Test (SDOIT). Dietary data were collected using a validated semiquantitative food frequency questionnaire. A total diet score (TDS) was calculated for intake of selected food groups and nutrients for each participant as described in the national dietary guidelines. Final scores ranged from 0 to 20; higher scores indicated closer adherence to dietary guidelines. After adjusting for all potential confounders, older adults with moderate/severe olfactory impairment (SDOIT score ≤ 3; lower scores indicate impairment) compared with those with no olfactory impairment had significantly lower adjusted mean (±SE) TDS, 9.09 (0.40) versus 9.94 (0.10), p = 0.04. Women with moderate/severe impaired olfaction (i.e., scored poorly on the odor identification test) compared with those with normal olfaction had significantly lower adjusted mean TDS, 8.87 (0.69) versus 10.31 (0.13), p = 0.04. No associations were observed between olfaction and TDS in men. Olfactory impairment in older women could signal an increased risk of poorer diet quality, defined as adherence to national dietary guidelines. Additional longitudinal studies are needed to confirm or refute the observed link between olfactory loss and overall patterns of food intake in older adults.