One of biggest obstacles for the development of terahertz (THz) science and technology is the lack of the THz sources, especially, when the requirement of the power reaches to kilowatts. Among all the existing sources, the gyrotron is the most promising one which can generate high-power THz radiation. Recently, a THz detection system is under development in the THz Research Center, University of Electronic Science and Technology of China, Chengdu, China, which is applied for detecting radioactive material. Thus, a 0.68-THz second harmonic gyrotron has been simulated and tested in the center. In this article, we present the theoretical study on the designed gyrotron based on the linear and nonlinear theories. The influences of the different electron beam parameters on the beam–wave interaction are analyzed. The mode competition based on the time-dependent multimode nonlinear theory has also been investigated in the designed gyrotron; it is shown that the operating mode can dominate the mode competition when the gyrotron is working at the optimized operating point. The experiment results show that the output power can reach 3.25 kW with the efficiency 9.26%, at frequency 0.679 THz, when the beam voltage is 39 kV, the beam current is 0.9 A, and the magnetic field is 12.84 T.