Thermoacoustic instability is a common problem in the operation of modern gas turbines. The prediction of thermoacoustic instability and the clarification of its mechanism are the research focus and difficulty in the gas turbine industry. As a response function of flame to acoustic disturbance, flame transfer function is a key parameter in the study of thermoacoustic instability. In this paper, based on the scaled adaptive simulation (SAS) model combined with the eddy dissipation concept (EDC) combustion model, the time-domain flow field data are processed by the system identification method, and the results of flame transfer function extraction are in good agreement with the experimental values. Then, the detailed derivation process of the low-order thermoacoustic network model (LOTAN) is given to capture the behavior characteristics of the acoustic wave in the thermoacoustic system. On this basis, the effects of acoustic boundary conditions and hysteresis time on the thermoacoustic instability of the combustion system are analyzed, and the relationship between the mode shape, pressure, vibration velocity phase, and thermoacoustic instability is explored. It is found that the phase relationship between pressure and vibration mode can be used to determine the thermoacoustic instability of the system. This is of great practical significance for determining the thermoacoustic instability of the system and clarifying the internal mechanism of its generation and provides theoretical support for the subsequent thermoacoustic instability control.