This paper presents a study of a novel nonlinear damper on vibration control of rotor system with variable damping by using a smart material- shear thickening fluid (STF), whose viscosity and damping will dramatically increase with shear rate. A STF damper was self-developed and dynamic model of the STF damper-rotor system was established by using the Bouc-Wen model as mechanical model of the smart material damper. The incremental harmonic balance (IHB) method combined with the arc-length method was presented to investigate the dynamic response of rotor system with hysteresis nonlinear damping force theoretically. Experimental study of STF damper on suppressing vibration of rotor system was carried out by using a Bently rotor test rig with two STF dampers. The damping effect under different damper installation direction, vibration frequency and amplitude were investigated theoretically and experimentally. Study shows that the nonlinear damper exhibits larger damping force and fuller hysteresis characteristics, so it has better effect on resonance energy absorption compared with the traditional linear damper; and its output damping force can increase dramatically with the change of frequency and amplitude. It is verified that the unbalance vibration, especially the resonance amplitude of rotor system can be effectively suppressed by nonlinear damping of STF damper. The study shows the feasibility and advantages of using the STF damper for potential applications of adaptive or semi-active vibration control of rotating machinery.