This research focuses on the nonlinear vibration control of a self-excited single-degree-of-freedom system. The integral resonant controller (IRC) is introduced to stabilize the unstable motion and suppress nonlinear oscillations of the considered system. The nonlinear dynamical equations that govern the vibratory behaviors of the proposed closed-loop control system are investigated using perturbation analysis, where loop delays have been included in the studied model. The system bifurcation behaviors have been visualized in both the two and three-dimensional spaces, and corresponding dynamical behaviors have been explored numerically using the bifurcation diagrams, Poincaré map, time-response, zero-one chaotic test algorithm, and frequency spectrum. The obtained analytical investigations revealed that the uncontrolled system can oscillate with one of four vibration modes depending on the excitation frequency, which are mono-stable periodic motion, bi-stable periodic motion, periodic and quasi-period motion, and quasi-periodic motion only. In addition, it is found that the existence of time delays in the control loop can either improve or degrade the control performance. Therefore, an objective function has been introduced to design the optimum control parameters. Based on the derived objective function, it is found that the performance of the proposed control strategy is proportional to the product of the control and feedback gains and inversely proportional to the internal loop feedback gain when the loop delays are neglected. Moreover, it is reported that the controller performance is a periodic function of the total sum of the loop delays. Accordingly, the optimal operating conditions of the time-delayed integral resonant controller have been explained. Finally, numerical validations for all obtained analytical results have been performed, where an excellent correspondence between the analytical and numerical investigations has been demonstrated.