To resolve the severe wear problems of steering engine and fin stabilizer, the nonlinear response characteristics of steering engine and fin stabilizer are studied by analyzing sea trial data of ship Yukun. The nonlinear response characteristics are summarized and applied to a rudder and fin hybrid control system. Decoupling, $$\hbox {H}_\infty$$ and $${\rm H}_2$$ controllers are designed to achieve low control frequency for it. Simulations show that all the three controllers can achieve excellent course-keeping and roll damping performances, and their control frequencies and amplitudes are consistent with those of nautical practice. It can be found that $$\hbox {H}_2$$ controller achieves the best response performance of course-keeping and energy saving performance for rudder engine; $$\hbox {H}_\infty$$ controller achieves the best response performance for roll damping; Decoupling controller has the best energy saving performance for fin stabilizer. When the Beaufort wind force scale is increased from 6 to 8 and model perturbation is involved, $$\hbox {H}_2$$ controller shows the best robustness of control performance and control input. In addition, the designed low-order controllers can reduce hardware cost for autopilot realization in nautical field.