Self-excited friction-induced oscillation, such as brake squeal, railway wheel squeal, chattering in machine tools, etc., is harmful in many mechanical systems. In the present study, the efficacy of acceleration feedback control with a second order filter is theoretically studied to mitigate friction-induced mode-coupling instability. Two different control strategies based on acceleration feedback are employed to control friction-induced mode-coupling instability of a two degrees-of-freedom linear system. Linear stability analysis near the equilibrium point of the system shows the efficacy of control laws to stabilize the system. The optimum filter parameters are determined by the pole crossover optimization method to attenuate the transient response as well as to achieve greater relative stability of the system. The Robustness study confirms the ability of the controller to stabilize the system in the presence of uncertainties in system parameters. Analytical results match closely with simulation results. Finally, nonlinearity in the form of stiffness is incorporated into the system for which the dynamics of the system show the presence of subcritical, supercritical Hopf bifurcation, and cyclic fold bifurcation. In the presence of nonlinearity, the feedback control is also able to reduce the range of instability of static equilibrium as well as the amplitude of the self-excited oscillation. The method of averaging is used to determine the stability and amplitude of the limit cycle. For the nonlinear system, the bifurcation diagram is constructed in AUTO and validated with results obtained from MATLAB SIMULINK and the method of averaging.