A methane/oxygen mixture is considered to be an appropriate propellant for many future rocket engines due to its practicality and low cost. To better understand the combustion instability in methane/oxygen-fed rocket engines, the spontaneous transverse combustion instability in a rectangular multi-element combustor (RMC) was analyzed both experimentally and numerically. Severe combustion instabilities occurred in the RMC during repeatable hot-fire tests. The physical mechanisms were systematically investigated through numerical simulations based on the stress-blended eddy simulation and flamelet-generated manifolds method with detailed chemical mechanisms (GRI Mech 3.0). The numerical results for the frequency spectrum and spatial modes agree well with the theoretical analysis and experimental data. The driven regions of the combustion instability were identified on both sides of the combustion chamber through a Rayleigh index analysis. The longitudinal pressure oscillations in the oxidizer post were found to be coupled with the transverse pressure waves in the combustion chamber and led to periodic oscillations of the mass flow rate of propellant. Moreover, the mixing was highly enhanced when the pressure wave interacted with walls of the combustion chamber. Therefore, a sudden release of heat occurred. The pressure oscillations were enhanced by pulsated heat release. A closed-loop system with positive feedback associated with periodic oscillations mass flow rate of the propellant, and sudden heat release, was believed to account for the present combustion instability.