People with epilepsy (PwE) suffer from progressive brain atrophy, which is reflected as neuroaxonal loss on the retinal level. This study aims to provide initial insight into the longitudinal dynamics of the retinal neuroaxonal loss and possible driving factors. PwE and healthy controls (HC; 18-55 years of age) underwent spectral domain optical coherence tomography at baseline and 7.0 ± 1.5 and 6.7 ± 1.0 months later, respectively. The change in retinal thickness/volume and annualized percentage change (APC) were calculated for the peripapillary retinal nerve fiber layer (pRNFL), the macular RNFL (mRNFL), the ganglion cell inner plexiform layer (GCIP), the inner nuclear layer, and the total macular volume (TMV). Group comparisons and multiple linear models with stepwise backward selection were performed to evaluate associations with demographic and clinical parameters. PwE (n = 44, 21 females, mean age = 35.6 ± 10.9 years) revealed a significant decrease in the pRNFL, mRNFL, GCIP, and TMV thickness or volume in the study interval. When compared to HC (n = 56, 37 females, mean age = 32.7 ± 8.3 years), the APC of the pRNFL (-.98 ± 3.13%/year) and the GCIP (-1.24 ± 2.56%/year) were significantly more pronounced in PwE (p = .01 and p = .046, respectively). Of note, atrophy of the mRNFL was significantly influenced by the number of antiseizure medications (ASMs; p = .047) and increasing age of PwE (p = .03). Contradictory results, however, were revealed for the impact of seizures. In epilepsy, progression of retinal neuroaxonal loss was already detectable at short-term follow-up. PwE who receive a high number of ASMs seem to be at risk for accelerated neuroaxonal loss, stressing the importance of well-considered and effective antiseizure therapy.