The runaway oscillation process of the pump-turbine in a high head pumped-storage power plant is usually unstable. The root cause of its instability is still unclear. In this paper, its internal mechanism and the improvement method were studied in depth. First, the flow characteristics in a model pump-turbine during the runaway process at four guide-vane openings (GVOs) were investigated by 3D transient numerical simulations. Then, the energy dissipation characteristics of different types of backflow vortex structures (BFVSs) occurring at the runner inlet and their impacts on the runaway stability were investigated by the entropy production theory. The results show that the location change of BFVSs between the hub side and the mid-span of the runner inlet around the no-load point leads to the sharp change in the energy dissipation rate, which makes the slope of dynamic trajectory positive and the runaway oscillation self-excited. If the occurrence of BFVSs at the hub side is suspended, the runaway process will be damped. Finally, the pump-turbine runner was improved to obtain a wider stable operating range.