In this study, the nonlinear dynamics of a dual-rotor system with active elastic support/dry friction dampers (ESDFDs) are investigated based on complex nonlinear modes (CNMs). The finite element method (FEM) combined with a full-3D friction model is introduced to construct the governing equation for the system. Additionally, the Craig–Bampton technique is applied to downscale the finite element model of the system. Based on the reduced order model (ROM), the nonlinear modal damping ratio of the target mode is employed to measure the dry friction damping performance of active ESDFD. The effects of the active ESDFD position, normal force, and tangential contact stiffness on the nonlinear modal damping ratio and modal frequency are analysed. Moreover, the softening characteristics of the active ESDFD are revealed, and the critical speed intervals of the active ESDFD/dual-rotor system are determined. Furthermore, by using the harmonic balance–alternating frequency/time domain (HB–AFT) method, the steady-state response of the system under unbalanced excitation is calculated. The accuracy and effectiveness of nonlinear modal analysis are validated based on the relationships between nonlinear modes and steady-state unbalanced responses. Conversely, the vibration mitigation effects of active ESDFD are determined by the unbalanced response amplitude. Additionally, the controllable region and optimal normal force for effective vibration control in the target mode are defined. Depending on the controllable region, a control strategy for turning on/off the optimal normal force is developed. The findings demonstrate that the developed control strategy enables the active ESDFD to significantly reduce the response amplitude of the dual-rotor system across various excitation levels, showing substantial potential for engineering applications.