Hippocampal atrophy is a well-known feature of age-related memory decline, and hippocampal subfields may contribute differently to this decline. In this cross-sectional study, we investigated the associations between hippocampal subfield volumes and performance in free recall and recognition memory tasks in both verbal and visual modalities in older adults without dementia. We collected MRIs from 97 (41 males) right-handed participants aged over 60. We segmented the right and left hippocampi into (i) dentate gyrus and cornu ammonis 4 (DG/CA4); (ii) CA2 and CA3 (CA2/CA3); (iii) CA1; (iv) strata radiatum, lacunosum and moleculare; and (v) subiculum. Memory was assessed with verbal free recall and recognition tasks, as well as visual free recall and recognition tasks. Amyloid-β and hippocampal tau positivity were assessed using [18F]AZD4694 and [18F]MK6240 PET tracers, respectively. The verbal free recall and verbal recognition performances were positively associated with CA1 and strata radiatum, lacunosum and moleculare volumes. The verbal free recall and visual free recall were positively correlated with the right DG/CA4. The visual free recall, but not verbal free recall, was also associated with the right CA2/CA3. The visual recognition was not significantly associated with any subfield volume. Hippocampal tau positivity, but not amyloid-β positivity, was associated with reduced DG/CA4, CA2/CA3 and strata radiatum, lacunosum and moleculare volumes. Our results suggest that memory performances are linked to specific subfields. CA1 appears to contribute to the verbal modality, irrespective of the free recall or recognition mode of retrieval. In contrast, DG/CA4 seems to be involved in the free recall mode, irrespective of verbal or visual modalities. These results are concordant with the view that DG/CA4 plays a primary role in encoding a stimulus' distinctive attributes, and that CA2/CA3 could be instrumental in recollecting a visual memory from one of its fragments. Overall, we show that hippocampal subfield segmentation can be useful for detecting early volume changes and improve our understanding of the hippocampal subfields' roles in memory.