It is well established that episodic memory declines with age and one of the primary explanations for this decline is an age-related impairment in the ability to form new associations. At a neural level, both the medial temporal lobe (MTL) and lateral prefrontal cortex (PFC) are thought to be critical for associative memory, and grey matter volume loss in these regions has been associated with age-related declines in episodic memory. While some recent work has compared the relative contributions of grey matter volume in MTL and PFC regions to item and associative memory, studies investigating the unique and shared contributions of age-related differences in the MTL and PFC to memory differences are still rare. In this study, we use a lifespan approach to examine the relationship between grey matter volume within substructures of the MTL and PFC on the one hand and item and associative memory on the other. To this end, we used data from over 300 healthy individuals uniformly spread across the adult lifespan from the Cambridge Centre for Ageing and Neuroscience (Cam-CAN) and tested the multivariate relationship between grey matter volumes and item/associative memory scores using canonical correlation analysis. We show that structures of the PFC alone predict memory performance better than either structures of the MTL alone or PFC and MTL combined. Moreover, our results also indicate that grey matter volume in the inferior frontal gyrus - pars opercularis, superior frontal gyrus, and middle frontal gyrus relates most strongly to memory (particularly associative memory, which loaded higher than item memory) and this effect persists when controlling for age and education. Finally, we also show that the relationship between frontal grey matter volume and memory is not moderated by age or sex. Taken together, these findings emphasize the critical role of the frontal lobes, and the control processes they subserve, in determining the effects of age on associative memory.