This article presents a comprehensive approach towards mitigation of four-wave mixing (FWM) induced crosstalk for a super-combined dense wavelength division multiplexing (SC-DWDM) system with in-line erbium-ytterbium doped fiber amplifier (EYDFA) + Raman hybrid optical amplifier (HOA). The performance of modified duo-binary return to zero (MDRZ) and differential phase-shift keying (DPSK) modulation formats employed on alternate channels of the 40-channel system have been compared considering equal (ECSA) and unequal channel spacing allocation (UECSA) schemes at a data rate of 40 Gbps. In the case of the UECSA scheme, the DPSK modulated channels have shown a reduction of FWM crosstalk components and intensities of up to 80% and 40 dB, respectively at a low bit error rate (BER) of 2.5 × 10−14. Different artificial neural network (ANN) techniques have been compared for gain spectrum estimation of HOA has been carried out based on input features of channel spacing, the number of channels and wavelength. Levenberg-Marquardt's (LM) model proves to be the best fit as compared to other ANN techniques. The k-nearest neighbours (KNN), support vector machine (SVM) and decision tree (DT) algorithms have been applied for the estimation of impairments based on root mean square error (RMSE) and coefficient of determination (R2) values as performance metrics. The best-performing model results applying an ANN approach with predicted R2 values of quality - factor (Q-factor) and noise figure (NF) of 0.74 and 0.56, respectively.