Hybrid power systems are now widely utilized in a variety of vehicle platforms due to their efficacy in reducing pollution and enhancing energy utilization efficiency. Nevertheless, the existing vehicle hybrid systems are of a considerable size and weight, rendering them unsuitable for integration into 25 kg compound-wing UAVs. This study presents a design solution for a compound-wing vertical takeoff and landing unmanned aerial vehicle (VTOL) equipped with an improved series hybrid power system. The system comprises a 48 V lithium polymer battery(Li-Po battery), a 60cc internal combustion engine (ICE), a converter, and a dedicated permanent magnet synchronous machine (PMSM) with four motors, which collectively facilitate dual-directional energy flow. The four motors serve as a load and lift assembly, providing the requisite lift during the take-off, landing, and hovering phases, and in the event of the ICE thrust insufficiency, as well as forward thrust during the level cruise phase by mounting the variable pitch propeller directly on the ICE. The entire hybrid power system of the UAV undergoes numerical modeling and experimental simulation to validate the feasibility of the complete hybrid power configuration. The validation is achieved by comparing and analyzing the results of the numerical simulations with ground tests. Moreover, the effectiveness of this hybrid power system is validated through the successful completion of flight test experiments. The hybrid power system has been demonstrated to significantly enhance the endurance of vertical flight for a compound-wing VTOL by more than 25 min, thereby establishing a solid foundation for future compound-wing VTOLs to enable multi-destination flights and multiple takeoffs and landings.