Introduction Decreasing retinal nerve fiber layer thickness (RNFL) has been associated with an increased risk of developing cognitive impairment in older adults. This may indicate that the integrity of retinal axons partially parallels that of the central nervous system. However, it is not clear from previous research whether RNFL thickness is associated with cognitive performance in cognitively normal adults. In this cross-sectional study, we used spectral-domain optical coherence tomography (SD-OCT) and detailed cognitive evaluations to determine the degree to which peripapillary RNFL (pRNFL) thickness is associated with cognition across the adult lifespan in a large community-based sample. Methods The data used for this investigation were obtained from 1306 participants of the on-going longitudinal Rhineland Study (age range = 30–95 years). During participants’ first visit, pRNFL thickness was measured using SD-OCT and neuropsychological assessments were completed as part of a larger battery. Cognitive performance was assessed in executive function, working memory, episodic memory and crystallized intelligence. Z-scores were calculated for continuous variables. We used linear regression analysis to quantify the strength of the relationships between pRNFL and cognitive test scores, expressed as difference in Z-score per SD increase in pRNFL. We performed three analyses: unadjusted; adjusted for age; adjusted for age, hypertension, refraction and visual acuity. Sensitivity analyses were conducted to determine whether the presence of cataracts or glaucoma influenced results. Results Participants with missing pRNFL (n = 50) or covariate (n = 47) data were excluded, leaving preliminary results available for 1209 participants, who had a mean age of 56 years (SD = 14 years) and comprised more women than men (56%). Overall, mean thickness of pRNFL was 99.91 μm (SD = 11.06 μm), and each additional year of participant age was associated with a thinning in pRNFL of 0.18 μm (95% CI = 0.13–0.23). In unadjusted models, thicker pRNFL was associated with better performance in executive functioning (0.15 SD, 95% CI = 0.09–0.21) and episodic memory (0.13 SD, 95% CI = 0.06–0.17), with worse performance in crystallized intelligence (0.08 SD, 95% CI = 0.02–0.14) and was not associated with working memory performance (0.03 SD, 95% CI = −0.02 to 0.09). These small effects largely disappeared upon adjustment for age and were no longer statistically significant (episodic memory: 0.04 SD, 95% CI = −0.01 to 0.10; crystallized intelligence: −0.04 SD, 95% CI = −0.10 to 0.02), except for the association of pRNFL and executive function (0.06 SD, 95% CI = 0.00–0.12). Further adjustment for hypertension, refraction and visual acuity did not change this latter relationship (0.07 SD, 95% CI = 0.01–0.12). The removal of participants with prevalent cataracts or a diagnosis of glaucoma did not affect the point estimate of the association of pRNFL with executive function, yet the 95% CI widened due to the smaller sample size (0.07 SD, 95% CI = −0.01 to 0.12; n reduction = 145). Conclusion In this exploratory investigation, we observed that pRNFL thickness was not associated with cognitive performance after adjustment for age, aside from a small association with executive function.