In this paper, we propose for the first time to apply probabilistic neural network (PNN) to fiber nonlinear compensation (NLC) in a high-speed coherent optical communication system that combines few-mode fiber (FMF) mode division multiplexing (MDM) with wavelength division multiplexing (WDM). By training on a small number of symbols, the PNN equalizer can compute the statistical distribution of the signal nonlinearity, make decisions on test symbols, and thereby compensate for the nonlinear impairments of the signals. To verify the effectiveness of the proposed scheme, we construct a MDM-WDM simulation system with 28 GBaud polarization division multiplexing (PDM) 16-quadrature amplitude modulation(QAM) and 64 GBaud PDM-64QAM, 6 spatial mode signals are transmitted over 11 wavelength channels with link lengths of 1500 km and 400 km, respectively. The results show that the PNN equalizer can make nonlinear judgments on received signals and effectively compensate the signal impairments caused by Kerr nonlinearity and device nonlinearity in a large launch power range after undergoing carrier phase recovery (CPR). Meanwhile, this scheme exhibits strong robustness against residual linear mode coupling (LMC), differential mode group delay (DMGD), and amplified spontaneous emission (ASE) noise. Finally, by the computational complexity comparison, it is proved that the complexity of the probabilistic neural network nonlinear compensation (PNN NLC) scheme is lower than that of digital back propagation (DBP) with comparable performance, which is on the order O(N).