This paper investigates a higher-order modified nonlinear Schrödinger equation with higher-order dispersion and self-steepening effects, which can be used to study the dynamics of asymmetric and steepened optical pulse transmission in optical fibres. The modulation instability of the plane-wave solution has been analysed, and the state transition of the rogue waves under the higher-order dispersion effect is presented. The findings demonstrate that the self-steepening effect may also produce an asymmetric unstable frequency band. Combined with the modulation instability, the rogue wave, rational soliton and mixed interaction of localized waves have a quantitative relationship with plane-wave parameters. Various exact solutions can be accurately located and obtained according to the generalized Darboux transformation. The asymptotic analysis of rational solutions demonstrates the state transition of higher-order rogue waves. This paper illustrates the significance of modulation instability for studying the integrable systems by the Darboux transformation. It is a new guidance to solve the difficulty of exact excitation of asymmetric localized waves in complex integrable systems. The results have prospective applications and references for the emergence, amplification and compression of asymmetric pulses in optical experiments.