The cellular instabilities of expanding hydrogen-propane-nitrous oxide spherical flames were experimentally and theoretically investigated at various equivalence ratios (ϕ=0.4–1.6), hydrogen fractions (XH=0–0.8), and initial pressures (Pu=1.1–1.5 bar). The experiments were conducted in a constant volume combustion chamber (CVCC) using high-speed schlieren technology to record the flame morphology, while the theoretical analysis was based on linear stability theory. The controlling parameters, critical conditions for the onset of instability, flame front oscillation intensity, logarithmic growth rate of perturbation, etc. were studied. The results indicate that the instability of premixed hydrogen-propane-nitrous oxide flames is dominated by hydrodynamic instability at XH≤0.6, while the competitive mechanism between thermal-diffusion and hydrodynamic instabilities is considered at XH=0.8. The oscillation intensity in the horizontal direction of the flame profile is more prominent than those in other directions, and with the increasing hydrogen fraction, the equivalence ratio at which the highest oscillation strength occurs shifts from ϕ=1.0 to ϕ=0.6. The trends of the experimental critical radius and Peclet number with each initial condition are completely identical to that of the theoretical value, but the experimental value is somewhat underpredicted. The three main reasons are proposed in the paper.Novelty and Significance Statement: This work is the first study on the cellular instabilities of hydrogen-propane-nitrous oxide flames by experiment and theory. It is under a small scale and reduced initial pressure condition that the cellular destabilization of hydrogen-propane flames is realized in the nitrous oxide atmosphere, while in the air atmosphere, the initial condition at which flame cellular destabilization occurs is harsher. Flame front oscillation strength, growth rate of disturbance, etc. were used to characterize quantitatively the degree of wrinkles and instabilities. Some novel findings were presented in the study. The present work is of great significance for the combustion community to understand deeply the cellular instabilities characteristic of expanding spherical flames supported by nitrous oxide, as well as for the engineering community to optimize the engine, propulsion, and energy supply systems, etc.