Background/Objectives: Vulnerability to sound distraction is commonly reported in older adults with dementia and tends to be associated with adverse impacts on daily activity. However, study outcome heterogeneity is increasingly evident, with preserved resistance to distraction also evident. Contributory factors may include individual differences in distractibility in older adulthood per se, and failure to consider the influence of how difficult a person found the test. Methods: We therefore measured distractibility in a group of older adults by comparing the performance of a primary visual task (Swansea Test of Attentional Control), which includes an adaptive algorithm to take into account how difficult a person finds the test under both no-sound and sound conditions. Results: Analysis revealed no significant difference in group mean performance between no-sound versus sound conditions [t (33) = 0.181, p = 0.858; Cohen’s effect size d = −0.028], but individual differences in performance both within and between sound and no-sound conditions were evident, indicating that for older adults, distracting sounds can be neutral, detrimental, or advantageous with respect to visual task performance. It was not possible to determine individual thresholds for whether sound versus no-sound conditions affected a person’s actual behaviour. Conclusions: Nevertheless, our findings indicate how variable such effects may be in older adults, which in turn may help to explain outcome heterogeneity in studies including people living with dementia. Furthermore, such within-group heterogeneity highlights the importance of considering a person’s individual performance in order to better understand their behaviour and initiate interventions as required.