This paper presents a numerical model for analysis of the aeroelastic stability of a thin cylindrical shell subjected to external supersonic airflow including consideration of the effect of geometric nonlinearity of the shell on the dynamic behaviour of the structure. The nonlinear stiffness matrix is derived based on a new formulation developed for thin cylindrical shells, which accounts for the shell curvature effect in the circumferential direction of the displacement field, and the effect of coupling between the different nonlinear modes on the dynamic behaviour of the shell. Nonlinear strain–displacement relationships are inferred from Novozhilov’s theory. The linear and nonlinear mass and stiffness matrices are calculated by adopting the displacement functions derived from exact solutions of linear Sander’s theory equilibrium equations for thin cylindrical shells. The classic finite element method is employed to derive the global mass and stiffness matrices. The aerodynamic damping and stiffness matrices are evaluated using the first–order potential (piston) theory. The initial stiffening effect due to radial pressure and/or axial loading is also taken into account. The governing equations of motion are derived using the Lagrange method and solved numerically with the help of a direct iterative method. Numerical studies are conducted to illustrate the effects of different parameters including internal pressure, nonlinear coupling, circumferential wave number, radius-to-thickness ratio, length-to-radius ratio and boundary conditions on flutter boundaries and on the nonlinear dynamic behaviour of the cylindrical shell at the onset of flutter and during the flutter stage. Good agreement is found between the results obtained using the presented approach and those published in the literature.