Introduction Physical function and cognition seem to be interrelated, especially in the oldest-old. However, the temporal order in which they are related and the role of brain health remain uncertain. Methods We included 338 participants (mean age 93.1 years) from two longitudinal cohorts: The UCI 90+ Study and EMIF-AD 90+ Study. We tested the association between physical function (Short Physical Performance Battery, gait speed and handgrip strength) at baseline with cognitive decline (MMSE, memory tests, Animal fluency, Trail Making Test (TMT) A and Digit Span Backward), and the association between cognition at baseline with physical decline (mean follow-up 3.3 years). We also tested whether measures for brain health (hippocampal, white matter lesion and gray matter volume) were related to physical function and cognition, and whether brain health was a common driver of the association between physical function and cognition by adding it as confounder (if applicable). Results Better performance on all physical tests at baseline was associated with less decline on MMSE, memory and TMT A. Conversely, fewer associations were significant but better scores on memory, TMT A and Digit Span Backward were associated with less physical decline. When adding measures for brain health as confounder, all associations stayed significant except for memory with gait speed decline. Discussion In the oldest-old, physical function and cognition are strongly related, independently of brain health. Also, the association between physical function and cognitive decline is more pronounced than the other way around, suggesting a potential for slowing cognitive decline by optimizing physical function.