Advanced age and brain damage have been reported to increase the propensity to gaze down while walking, a behavior that is thought to enhance stability through anticipatory stepping control. Recently, downward gazing (DWG) has been shown to enhance postural steadiness in healthy adults, suggesting that it can also support stability through a feedback control mechanism. These results have been speculated to be the consequence of the altered visual flow when gazing down. The main objective of this cross-sectional, exploratory study was to investigate whether DWG also enhances postural control in older adults and stroke survivors, and whether such effect is altered with aging and brain damage. Posturography of older adults and stroke survivors, performing a total of 500 trials, was tested under varying gaze conditions and compared with a cohort of healthy young adults (375 trials). To test the involvement of the visual system we performed spectral analysis and compared the changes in the relative power between gaze conditions. Reduction in postural sway was observed when participants gazed down 1 and 3 meters ahead whereas DWG towards the toes decreased steadiness. These effects were unmodulated by age but were modulated by stroke. The relative power in the spectral band associated with visual feedback was significantly reduced when visual input was unavailable (eyes-closed condition) but was unaffected by the different DWG conditions. Like young adults, older adults and stroke survivors better control their postural sway when gazing down a few steps ahead, but extreme DWG can impair this ability, especially in people with stroke.