Changes in retinal thickness and the morphology of fluid provide valuable diagnostic information for Macular Edema (ME). Optical Coherence Tomography (OCT) is a pivotal diagnostic tool for ME, and the accurate segmentation of OCT retinal images aids ophthalmologists in monitoring disease progression. However, the unpredictable nature of ME’s occurrence poses significant challenges for existing methods in achieving precise segmentation of both retinal layers and fluids simultaneously. In this study, a novel U-shaped encoder-decoder architecture named GD-Net, is introduced for simultaneously segmenting retinal layers of diverse thicknesses and fluids. Specifically, a novel Fast Fourier Encoder is integrated within the encoder branch to capture spectral domain features overlooked by the spatial encoder. Additionally, Multi-scale Graph Convolution module is inserted between the encoder and decoder branches to leverage retinal layer topology for spatial reasoning. GD-Net is further optimized using a joint loss function, which includes a weighted version of region segmentation loss and contour loss, effectively mitigating the issues of blurred fluid edges and internal holes. The method was validated on three publicly available datasets: DUKE DME, Peripapillary OCT, and RETOUCH, achieving overall mean Dice score of 0.839, 0.826, and 0.872, respectively, outperforming the other state-of-the-art techniques. GD-Net demonstrated consistent high-precision segmentation on B-scans from different devices, proving its robustness and surpassing other methods in terms of visual quality, thus offering substantial support for clinical ophthalmologists in diagnosing ME. The code and weights are publicly available at: https://github.com/DBook111/GD-Net.