The objective of this research is to design a robust control system for trajectory tracking of the under-actuated Rotary Inverted Pendulum (RIP). The focus is on achieving semi-global asymptotic stability in the absence of knowledge about the RIP geometric and inertia parameters. The control design commences by formulating differential servo constraint dynamics (VCD) that encapsulates the control objectives for the RIP. The baseline generalized dynamic inversion (GDI) control law is derived by inverting the VCD for the control voltage input using the Moore–Penrose generalized inverse. To enhance robustness against modeling uncertainties and exogenous disturbances, a sliding mode control (SMC) element is integrated within the GDI control system, resulting in the robust GDI (RGDI) control system. Furthermore, an adaptive estimator that is based on Radial Basis Function Neural Networks (RBF-NN) is adopted to eliminate the dependency of the RGDI control system on the RIP mathematical model. The weighting matrices of the RBF-NN are updated with the aid of a Lyapunov control function. The proposed control system offers robust solution and guarantees semi-global asymptotic stability to the unstable equilibria of the RIP dynamics in the absence of a mathematical model. The proposed control system’s performance is evaluated through computer simulations and experimental tests on a real RIP system. Comparative analyses with traditional SMC and linear quadratic control methodologies are conducted to show the effectiveness of the proposed approach.