Guide vanes (GVs) are installed between the runner and the stay vanes for flow guidance and discharge regulation in reversible pump-turbines. The unstable torsional self-excited vibration of the guide vane (GV) may occur when running at small guide vane opening angles during the transient operations involving pump flow. In addition, the double-stage radial vanes may induce complex flow in the vanes and influence the stability of torsional self-excited vibration of the guide vane. In this study, numerical simulations were conducted at small guide vane opening angles in pump mode for two different guide vanes based on the three-dimensional computational fluid dynamics (CFD) method. The flow regime with a deflection was formed on the trailing edge with a circle in the vaneless region that rotated reversely against the runner rotation when operating at smaller guide vane opening angles for both of the two guide vanes. Based on this, the coupling simulations based on the CFD method with a single-degree-of-freedom (1DOF) oscillator were carried out under these operating conditions. Two flow types were formed at small opening angles when adopting different inlet boundary conditions. The results showed the flow regime with a deflection on the trailing edge may aggravate the instability of torsional vibration when applied as an initial flow field. Moreover, the vibration instability of the torsional self-excitation for two guide vanes was analyzed, showing that modifying the profile of guide vane airfoil is an efficient and reliable approach for weakening the torsional vibration instability.