Gyrotron oscillators are one of promising sources to generate high power terahertz radiation. To achieve high power at high frequency, gyrotrons have to operate at high order modes to alleviate the problem of wall heating and the miniaturization of interaction structures at these frequencies. Due to high order modes, the mode spectrum is very dense and the electron beam is likely to interact with several modes. Mode competition is a major hurdle in achieving stable and efficient operation of a gyrotron at cyclotron harmonics. The gradually tapered complex cavity could suppress mode competition by locking a pair of operating modes in two single cavities. Starting from the general transmission line equations with an electron beam, self-consistent nonlinear simulations on the second harmonic complex cavity gyrotron operating at 0.42 THz are presented. The influences of the electron beam parameters on the interaction efficiency are analyzed. The experimental results show that the designed complex cavity gyrotron is operationally stable at TE17.4 with pulse output power of 22.25 kW when the beam current is 4.8 A, beam voltage is 45 kV, and applied magnetic field is 8.185 T. The corresponding frequency and interaction efficiency are 421.60 GHz and 10.27%, respectively.