Studies investigating age-related functional differences associated with source memory have recently focused on the importance of clarifying the relationship between effects of age and performance on memory-related brain activations. One methodological challenge has been in discriminating between effects of age on memory-related brain activations that are independent from age-related differences in performance. In the current study, event-related functional magnetic resonance imaging (fMRI) was used to identify brain activity during spatial and temporal source encoding and retrieval across the adult lifespan. We used multivariate behavior partial least squares (B-PLS) to identify patterns of brain activity during source encoding and source retrieval that were associated with age and/or retrieval accuracy. The PLS analysis identified three significant effects. The first effect indicated that encoding and retrieval activity in fusiform, middle occipital–temporal and inferior parietal cortices increased with age and decreased with performance. The second effect showed that dorsolateral prefrontal cortex and limbic activity increased with age at encoding, and increased with performance at retrieval. The third effect indicated that activity in right ventrolateral prefrontal and bilateral hippocampus increased with age during retrieval and was differentially related to performance during encoding versus retrieval. We conclude that although some age-related differences in brain activity observed during source encoding and retrieval are associated with individual differences in performance, age-related differences in prefrontal and hippocampal areas exhibit more complex patterns of interactions between age, performance, and phase-related activity.