We mainly investigate a coupled system of the generalized nonlinear Schrödinger equation and the Maxwell–Bloch equations which describes the wave propagation in an erbium-doped nonlinear fiber with higher-order effects including the forth-order dispersion and quintic non-Kerr nonlinearity. We derive the one-fold Darboux transformation of this system and construct the determinant representation of the n-fold Darboux transformation. Then the determinant representation of the nth new solutions (E[n], p[n], η[n]) which were generated from the known seed solutions (E, p, η) is established through the n-fold Darboux transformation. The solutions (E[n], p[n], η[n]) provide the bright and dark breather solutions of this system. Furthermore, we construct the determinant representation of the nth-order bright and dark rogue waves by Taylor expansions and also discuss the hybrid solutions which are the nonlinear superposition of the rogue wave and breather solutions.