Abstract Nonlinear long haul optical transmission systems (NLOTSs) are a highlighted research area in the telecommunication sector. The importance of NLOTSs is increased, by reason of system volume, number of channels and transmission span demands. To achieve the current goals in the telecommunication industry, the system faces remedies called refractive index related nonlinearities (RIrNs). These nonlinearities cause performance discreditation of NLOTSs. In this paper, RIrNs are estimated and treated for NLOTS. This is achieved by applying modulation format indicators and amplitude deviation analysis. Channel spacing, input power, system capacity, fiber length, and quantity of channels parameters are used with reference to optical signal-to-noise ratio (OSNR) and bit error rate (BER) to simulate the preferred NLOTS model.