The effects of two-level resonant atoms and self-steepening effect on modulation instability in an erbium fiber is studied by considering the nonlinear Schrödinger Maxwell–Bloch (GINLS–MB) with generalized external potentials. By applying linear stability analysis (LSA), we examine the comparison between MI gain spectra for both with and without erbium effects. We depict certain significant results by graphically illustrating the 2D and 3D figures through suitably selecting the parameters. In addition, we provide a comparison table for adopting parameters. Obtained remarkable results are useful for the study of soliton stability in nonlinear optical communication system. Also, our investigations shows that the gain spectrum mainly influenced by erbium atoms and self-steepening term. Especially, symmetric manner of gain spectrum becomes as asymmetric due to the impact of self-steepening term. The theoretical findings carried out in this work will be beneficial for experimental observation of MI process in an erbium doped fiber.