Field-reversed theta-pinch (FRTP) is commonly used to form high-performance field-reversed configuration (FRC) plasmas, and it is employed for the initial plasma formation in the HFRC device. The high-voltage pulsed power supply which is used to feed the θ-pinch coils in HFRC has already demonstrated its importance for initial plasma formation. This paper presents two attempts to improve the performance of the high-voltage pulsed power supply. To reduce the space occupation and increase the accuracy of the triggers of hydrogen thyratron, a newly designed high-voltage floating trigger is proposed in this paper. Experimental tests show that the optimized trigger can work stably under a floating voltage of 35 kV with a voltage rising rate of about 12 kV/μs and a total jitter time of less than 60 ns. Besides, a high-voltage laminated busbar is proposed to reduce the stray parameters. Experimental tests show that it can withstand a voltage of 34 kV. The peak current and frequency of each branch of the power supply are increased by at least 10%. Tests of the entire power supply show that the power supply can work stably under a voltage of 34 kV. The oscillation frequency of the Pre-Ionization (PI) branch can reach 152.0 kHz and the peak current of the Main branch can reach 78.3 kA. The results show the validity and reliability of the improvements.