The excitation system of third harmonic excitation brushless synchronous generator (HESG) can improve the voltage stability and is widely used in small and medium generator set. In order to establish the model of harmonic brushless excitation system, the formula of third harmonic EMF is derived. Then, based on the traditional synchronous generator model, the influence of harmonic windings is counted in. Finally, the simulation model of closed-loop excitation system is analyzed with the automatic voltage regulator (AVR) as the core. On the basis of theoretical analysis, simulation and comparative analysis are carried out according to parameters from actual system. The results show that compared with normal synchronous generator, HESG has the advantage of excellent dynamic characteristics and automatic voltage adjustment.