The unstable combustion problem in small-sized solid rocket engines with a large aspect ratio is so complicated that its causes remain unclear. In this study, the coupled vibration between the sound field and shell in the engines was proposed as a possible cause. A solid rocket engine structure was abstracted into a multilayer thin-walled cylindrical cavity in this study, followed by the theoretical calculation and simulation calculation of its inherent frequency. Next, a thin-walled cylindrical cavity fluid-solid coupling experimental platform with the function of modal measurement was established to verify the accuracy of simulated modes for the shell structure and acoustic cavity. Then, the mode of the finite element model (FEM) for the solid rocket engine was theoretically calculated and simulated, accompanied by finite element calculation and experiment of the acoustic mode of the internal acoustic cavity. Subsequently, the engine mode was compared with the acoustic mode of the internal acoustic cavity. On this basis, a new cause for the damage and disintegration of the solid rocket engines in the final working stage was revealed. Moreover, a brand-new idea of inhibiting the pressure oscillation-induced unstable combustion in the solid rocket engines was put forward.