Excessive vibrations induced by intrinsic internal friction in spline joints critically impact the reliability and safety of high-speed rotating machinery. To address this concern, this paper presents experimental and numerical research on a flexible rotor system subjected to frictional contact within the spline joint. Experiments are conducted on a scaled rotor test rig designed to emulate the tail rotor drive shaft of helicopters, aiming to explore vibration behaviors under various operating conditions. A reduced-order mathematical model based on the test rig, considering the multi-interface friction of the spline joint and deformation coordination with a flexible coupling, is formulated. The numerical study, which incorporates both stability analysis and transient nonlinear dynamics, is then presented to quantitatively evaluate the occurrence conditions and excitation mechanisms leading to friction-induced instabilities. A parameter study is further conducted to elucidate the roles of various parameters in shaping self-excited vibration behaviors. The results reveal that the cross-coupling of frictional forces within the spline joint acts as the driving force of instabilities at supercritical speeds, with the resulting self-excited responses dominated by the frequency of the first critical speed. The study also highlights the necessity of integrating experimental and numerical analysis to better predict and understand the self-excited vibration phenomena in the spline-shaft systems.