Abstract The ability of native communities to maintain invasion resistance in the face of environmental changes is crucial for biodiversity conservation. However, previous studies have primarily focused on the inhibitive effects experienced by invaders, providing limited insights into the ultimate outcome of resistance. In this study, we integrated modern coexistence theory into biotic resistance research to investigate the impact of nutrient enrichment on the resistance of Phragmites australis marshes to Spartina alterniflora invasion in the Yangtze estuary saltmarshes of China. Our results demonstrated that under non‐enriched conditions, successful invasion resistance was facilitated by stable coexistence between native and invasive species in the field. This prediction of invasion resistance aligned with the distribution dynamics of the two species in the Yangtze estuary saltmarshes over the past two decades. However, nutrient enrichment was likely to lead to a fundamental shift in their coexistence and ultimately, the failure of resistance. Synthesis and applications. Integrating modern coexistence theory into biotic resistance studies advances the assessment of invasion resistance by shifting from quantifying relative strength to predicting explicit resistance outcomes. Ecosystem managers can draw explicit conclusions about the potential establishment and impact of invaders by analysing observational or experimental data within the framework of modern coexistence theory. This information aids in identifying the most efficient strategies for addressing invasive species.