We have designed and experimentally demonstrated the operation of a novel quasioptical gyrotron oscillator based on an overmoded confocal waveguide cavity. This cavity effectively suppresses undesired modes, and therefore has extremely low mode density. Stable single-mode, single-frequency operation was achieved in the TE/sub 06/ mode at 136 GHz. A peak RF output power of 66 kW, corresponding to an efficiency of 18%, was measured. By varying the cavity magnetic field, high-power generation was observed at 136 GHz in the TE/sub 06/ mode and at 114 GHz in the TE/sub 05/ mode. These frequencies correspond to the high Q modes of the confocal resonator. The low Q modes were either weak or not observed. In this paper, we will review the design procedure for this cavity and present experimental data verifying its effectiveness in reducing the number of modes that can be excited. The confocal waveguide could also be used in high-power, gyro-TWT amplifiers to provide greater operating stability and bandwidth, especially in an overmoded waveguide structure.