Abstract Modern communication generation for high data rate requirement is fifth generation. The fifth generation has many advantages like low delay, high spectrum availability, high reliability, low jitter, and more capacity. To combat with the high capacity and high data rate requirement, optical fiber helps in the backhaul of 5G network. When fiber is used as a mode of propagation, many nonlinearities arises. This nonlinearity effect in the optical fiber communication is one of the most undesirable phenomena for the modern communication system which results in harmonic distortion, intermodulation distortion, phase distortion, and adjacent channel interference, etc. In most fiber optic communication systems, the major fiber nonlinear effect is the Kerr nonlinearity and Scattering effect that produced due to the variation of refractive index due to signal intensity. To minimize the effect of nonlinear distortion, some techniques are used which considerably improve the transmission capacity by reducing or compensating the effect of nonlinear distortion along with minimal infrastructure modifications and minimum cost for implementation. To overcome the effect of nonlinearity, fiber nonlinearity mitigation techniques are preferred to achieve the significant performance gains. Mitigation techniques can be implemented to improve and optimize the performance of optical communication network. In this paper, a brief description of the fiber nonlinearity effect and a review of various fiber nonlinearity compensation techniques are described.