A permanent magnet spherical motor (PMSpM) with three degrees of freedom rotation characteristics in the rigid rotor has broad application prospects. But the adding flexible material will inevitably produce end jitter issue in the process of the rigid-flexible coupling system (RFCs) movement. To solve the above problem, a fractional order non-singular fast terminal sliding mode control method with a disturbance observer was proposed for the rigid-flexible coupling system of a permanent magnet spherical motor (RFCs-PMSpM). Firstly, a dynamic model of RFCs-PMSpM was established by using the Hamilton principle and Euler–Bernoulli beam theory. Secondly, the influences on the end jitter were discussed from the perspectives of load mass, material parameters, and driving torque of the flexible shaft. The sensitivity analysis is carried out, and the key factors affecting the jitter are obtained. Then, a fractional order non-singular fast terminal sliding mode controller based on a nonlinear disturbance observer (NDO-FONFTSMC) is proposed. The stability of the closed-loop control system was proved by the Lyapunov method. Finally, the effectiveness of the proposed jitter suppression strategy was validated and compared with existing approaches, which also provides an important reference for the future application of RFCs-PMSpM in high-precision industry.