The present article focuses on suppressing mode-coupling instability in frictional systems, which can cause unwanted vibrations such as squeal in braking systems. Mode-coupling instability occurs when two or more modes of a system approach each other. This article proposes the use of resonant velocity feedback control to eliminate friction-induced oscillations generated by mode-coupling instability. A two degree-of-freedom linear model is used to capture the instability due to friction-induced mode coupling. The efficacy of the control is demonstrated through linear stability analysis, and by optimizing control parameters using pole crossover method to minimize the transient time of the response. The robustness analysis demonstrates the ability of the control to effectively mitigate the instability even under parametric perturbations. The controlled system is simulated in MATLAB Simulink to validate analytical results. The effects of time delay, which is commonly present in feedback systems, are also investigated. Finally, the effectiveness of the control is studied in the presence of nonlinearity in the system. The nonlinear dynamics are analyzed by creating bifurcation diagrams with different bifurcation parameters.