Tau pathology is recognized as a primary contributor to neurodegeneration and clinical symptoms in Alzheimer's disease (AD). This study aims to localize the early tau pathology in cognitively normal older people that is predictive of subsequent neurodegeneration and memory decline, and delineate factors underlying tau-related memory decline in individuals with and without β-amyloid (Aβ). A total of 138 cognitively normal older individuals from the Berkeley Aging Cohort Study underwent 11 C-Pittsburgh Compound-B (PiB) positron emission tomography (PET) to determine Aβ positivity and 18 F-Flortaucipir (FTP) PET to measure tau deposition, with prospective cognitive assessments and structural magnetic resonance imaging. Voxel-wise FTP analyses examined associations between baseline tau deposition and longitudinal memory decline, longitudinal hippocampal atrophy, and longitudinal cortical thinning in AD signature regions. We also examined whether hippocampal atrophy and cortical thinning mediate tau effects on future memory decline. We found Aβ-dependent tau associations with memory decline in the entorhinal and temporoparietal regions, Aβ-independent tau associations with hippocampal atrophy within the medial temporal lobe (MTL), and that widespread tau was associated with mean cortical thinning in AD signature regions. Tau-related memory decline was mediated by hippocampal atrophy in Aβ- individuals and by mean cortical thinning in Aβ+ individuals. Our results suggest that tau may affect memory through different mechanisms in normal aging and AD. Early tau deposition independent of Aβ predicts subsequent hippocampal atrophy that may lead to memory deficits in normal older individuals, whereas elevated cortical tau deposition is associated with cortical thinning that may lead to more severe memory decline in AD. ANN NEUROL 2024;95:249-259.