To alleviate subsynchronous resonance in doubly fed induction generator–based wind farm, this study devises a robust nonlinear control for a rotor-side converter based on a fractional-order sliding mode controller. The designed fractional-order sliding mode controller is able to realize finite-time control and reduce the control time compared with terminal sliding mode control, which contributes to the faster mitigation of subsynchronous resonance. Impedance-based analysis and transient simulation are carried out to evaluate the performance of the fractional-order sliding mode controller compared with terminal sliding mode control and subsynchronous damping control. Simulation results verify that the fractional-order sliding mode controller is able to damp SSR within shorter time and effectively reduce the fluctuation range of a system’s transient responses under various working conditions of compensation degrees and wind speeds. Furthermore, the fractional-order sliding mode controller enhances the robustness under external disturbance and parametric uncertainty, ensuring safe operation of the practical wind farm.