In this paper, static and dynamic characteristics of a labyrinth seal were studied by fluid–solid–thermal coupling analysis (FSTCA). Deformation and stress distribution of nonmetallic labyrinth seals (i.e., PEEK and PEEK-CA30) were further analyzed to investigate the effects on leakage and rotor stability compared to that of metallic material (i.e., AL alloy). The results show that a higher pressure drop of airflow at the last tooth along the airflow direction suggested a more remarkable energy conversion to kinetic energy. Airflow velocity at the tooth root was obviously lower than that at the tooth tip because of vortex in tooth cavity. Maximum deformation of the seal occurred at tip of the first tooth, while equivalent stress mainly distributed at the root. Both axial and radial deformations led to deformations of seal cavities and upward movements of the teeth, resulting in greater radial clearances. PEEK-CA30 had the lower radial and axial maximum deformations compared to PEEK and AL alloy, which in turns caused a lower leakage with increasing inlet temperature and inlet/outlet pressure ratio and a higher effective damping coefficient under different vortex frequencies, indicating a better seal performance and a better stability of rotor system. The equivalent stress for AL alloy deeply depended on inlet temperature of airflow due to higher elasticity modulus, higher thermal conductivity and higher thermal expansion coefficient, which made the seal more likely to fail under higher inlet temperature.