The ports of a three-phase voltage source converter have multiple frequency electrical parameters, which makes HVDC systems with these converters present multiple input multiple output (MIMO) characteristics. The generalized Nyquist criterion of the MIMO model is often used to analyze the system stability. However, this model is more difficult to obtain than the single-input single-output (SISO) model, and the stability margin and the oscillation frequency band are difficult to capture. To solve these problems, this paper took the two-terminal HVDC transmission system as the research object to derive a SISO model that can be measured by a port. Moreover, this study also proposed a method for analyzing the system stability through envelope SISO open-loop gains. Finally, the key port and parameters affecting the system stability were analyzed according to the stability margin, which provided theoretical support for the design of the system parameters. The established model and the proposed analysis method were verified in a MATLAB/Simulink simulation model.