The article applies the robust generalized dynamic inversion control methodology to the problem of stabilizing upright equilibrium configuration of the under-actuated rotary inverted pendulum system while tracking rotary motion of the actuated arm. The proposed robust generalized dynamic inversion control law comprised equivalent and switching control parts. The equivalent control part works to enforce a virtual constraint dynamics of the controlled state variables by means of Moore–Penrose generalized inversion. The switching control part is of the sliding mode type, and it improves robustness against unmodeled system dynamics, parametric uncertainties, and external disturbances. The robust generalized dynamic inversion control design on the linearized model of the under-actuated rotary inverted pendulum is shown to guarantee semi-global asymptotically stable tracking performance. Numerous computer simulations and experiments are conducted on the Quanser rotary inverted pendulum system, revealing that the proposed algorithm has better convergence and tracking performance than conventional sliding mode and generalized dynamic inversion control strategies when both are applied separately.