Increasing the dimensions of high power microwave devices is an efficient method to improve the power capacity. However, an overmoded structure usually results in mode competition and a low beam-wave conversion efficiency. In this paper, a multi-mode operation mechanism is used to avoid mode competition and increase the efficiency. The calculation results of nonlinear theory of beam-multimode interaction show that the optimized conversion efficiency is up to 48% when TM01 mode, TM02 mode, and TM03 mode are all considered. As only the TM01 mode, TM02 mode, or TM03 mode is taken into account independently, the corresponding efficiency is 38%, 22%, or 20%. Based on this, a multi-mode relativistic backward wave oscillator is proposed with the ratio of the mean diameter of the slow wave structure (SWS) to the wavelength of the output microwave to be 3.5. The non-uniform SWS is used to increase the beam-wave conversion efficiency, and a combined reflector is adopted to reflect partial of the mixed microwave modes and make the device compact. The particle-in-cell simulations show that as the diode voltage is 1.1 MV, the beam current is 22.8 kA, and the external magnetic field is 0.76 T, the conversion efficiency is 45%, and the output microwave of 11.3 GW is the mixed modes of TM01 mode, TM02 mode, and TM03 modes with the corresponding power ratio of 74%, 7%, and 19%, respectively.