The optimization design of high-frequency transformer (HFT) is a multi-objective optimization problem owing to the mutual constraints between design parameters such as power density, efficiency and temperature rise, and traditional design methods often face difficulties in balancing and selecting multiple design parameters. Therefore, this paper derives the structural parameters of HFT, calculates the magnetic core losses and winding losses considering temperature effects, and establishes a 7-node thermal network model. Selecting power density and efficiency as optimization objectives, the HFT is optimized based on multi-objective particle swarm optimization algorithm and non-dominated genetic algorithm, and compared with traditional free parameter scanning method to analyze the advantages of intelligent optimization algorithm. Further, the influence of temperature rise calculation methods on the optimization design results of HFT is compared and analyzed. Finally, a nanocrystalline HFT with 17 MW/m3 power density and 99 % efficiency and an optimization scheme of HFT with 26 MW/m3 power density and 99.2 % efficiency are designed based on the proposed optimization design processes. The effectiveness of the calculation model and optimization method is verified through experimental measurement, and the essential principle of improving the HFT performance through optimization methods is explored through finite element simulation, providing theoretical reference and data support for the optimization design of nanocrystalline HFT.