This study aimed to examine the thicknesses of the ganglion cell layer (GCL) and peripapillary retinal nerve fiber layer (RNFL) in eyes with resolved macular edema (ME) in branch retinal vein occlusion (BRVO) and determine their relationship with visual acuity (VA). This retrospective observational case–control study included 57 eyes of BRVO patients with resolved ME after treatment. The macular GCL thickness, peripapillary RNFL thickness, and central macular thickness (CMT) measured on swept-source optical coherence tomography scans with the contralateral eyes used as controls were evaluated. The mean CMT was 270.48 ± 32.7 μm; the mean RNFL thickness was 105.46 ± 25.94 μm in BRVO eyes. Although the average RNFL thickness was decreased in BRVO eyes compared to unaffected eyes, there was no significant difference between the groups. However, the temporal and nasal RNFL thicknesses were significantly different between the groups. The mean affected quadrant had a significantly thinner GCL compared to the corresponding opposite unaffected quadrant (p = 0.02). Final VA was significantly correlated with nasal and middle GCL thicknesses in the affected area (r = −0.512, p = 0.003 and r = −0.537, p = 0.001, respectively); no correlation was found between the average RNFL thickness and mean CMT. The peripapillary RNFL and GCL thicknesses of the affected area were reduced in BRVO eyes compared to unaffected eyes. VA significantly correlated with nasal and middle GCL thicknesses in the affected area. Inner retinal damage occurring in patients with ME secondary to BRVO may be related to the visual prognosis.