<sec>The mode competition in an overmoded relativistic backward wave oscillator is studied through theoretical analysis and three-dimensional particle-in-cell simulation in this work. Based on the quality factor and coupling impedance, the mode selection for a TM<sub>02</sub> mode backward wave oscillator is achieved, and its output power and magnetic field strength are optimized in the simulation.</sec><sec>The quality factor is related to the group velocity and end reflection of each mode. The dispersion curves of some non-axisymmetric modes are very close, and the group velocities are basically equal. Therefore, the end reflection needs considering to distinguish between the quality factors of different modes. In frequency domain simulation, analyzing the quality factor of each mode by using the <i>S</i><sub>11</sub> parameter curve can avoid calculating the end reflection.</sec><sec>The three-dimensional simulation results show that the coupling impedance and quality factor jointly affect the operating mode. When the coupling impedance advantage of the working mode is not obvious, changing the resonant frequency of the high-frequency structure can affect the beam-wave interaction process, thereby changing the excitation mode. When the advantage is obvious, the beam-wave interaction of the excitation mode will not be destroyed by the resonant mode, and other modes of microwave output mainly come from the conversion of the same frequency modes. Due to the constant dispersion curve, the effect of resonance on the mode is essentially the effect of the quality factor on the mode dominated by the end reflection.</sec><sec>The insensitive parameters and the electron beam radius obtained from the simulation are used as the optimal parameters, and the automatic optimization algorithm is used in combination with the two-dimensional simulation to perform multi-objective optimization design in the above device. The final output power of the backward wave oscillator reaches 534 MW, with an efficiency of 23.64%, an increase of 221.7% compared with the efficiency of the original device. The device operating mode remains stable, with a power ratio of TM<sub>02</sub> mode reaching 94.95%.</sec>