This work is to investigate the dynamic instability and unsteady heat and mass transfer mechanism of Ammonium perchlorate/Hydroxyl-terminated polybutadiene (AP/HTPB) propellant in the chamber of solid rocket motor (SRM). The environment of exponential depressurization is imposed on the heterogeneous propellant combustion to simulate the actual working condition of SRM. The microstructure of AP/HTPB propellant is generated by a code based on the sequential algorithm. The combustion model consists of gas reaction, heat conduction in the solid phase, and the non-planar moving surface. A quenched model under the rapid pressure decay is established. It is found that the rapid depressurization and the change of surface morphology are responsible for the combustion fluctuation. Finally, quenched times of different propellant packs are calculated. It is found that different AP particle positions in the propellant pack will also cause a change in the quenched time.