Electron cyclotron maser interactions at terahertz (THz) frequencies require a high-order-mode structure to reduce the wall loss to a tolerable level. To generate THz radiation, it is also essential to employ cyclotron harmonic resonances to reduce the required magnetic field strength to a value within the capability of the superconducting magnets. However, much weaker harmonic interactions in a high-order-mode structure lead to serious mode competition problems. The current paper addresses harmonic mode competition in the gyrotron backward wave oscillator (gyro-BWO). We begin with a comparative study of the mode formation and oscillation thresholds in the gyro-BWO and gyromonotron. Differences in linear features result in far fewer “windows” for harmonic operation of the gyro-BWO. Nonlinear consequences of these differences are examined in particle simulations of the multimode competition processes in the gyro-BWO, which shed light on the competition criteria between modes of different as well as the same cyclotron harmonic numbers. The viability of a harmonic gyro-BWO is assessed on the basis of the results obtained.