Wide and rapid Spartina alterniflora invasion has threatened the sustainability of coastal wetlands of China, and has a significant impact on sediment biogeochemical cycle. However, the effects of exotic Spartina alterniflora invasion on sediment dissimilatory nitrates (NOx−) reduction processes and mechanisms underlying these pathways in mangrove wetlands have not been well studied. Here, denitrification (DEN), anaerobic ammonium oxidation (ANA) and dissimilatory nitrate reduction to ammonium (DNRA) rates, contributions (DEN%, ANA%, and DNRA%), associated functional genes (nirS, anammox 16S rRNA and nrfA) and controlling factors were investigated in surface sediments (0–5 cm) in mangrove and Spartina alterniflora communities and ecotone of the Zhangjiang Estuary, China. Significant spatial differences in DEN, DNRA, DEN%, and DNRA% were found among different plant communities (p < 0.05 for all). DEN is the dominant pathway of NOx− reduction processes in Spartina alterniflora community (87–89 %) and ecotone (75–78 %), while DEN (41–51 %) and DNRA (36–51 %) compete with each other in mangrove community. Organic matter was the most dominating factor controlling DEN and DNRA rates. Without accounting for confounding factors, Spartina alterniflora invasion significantly reduced the nitrogen retention index (NIRI), causing a large amount of nitrogen (N) loss in wetlands. We estimated that Spartina alterniflora invasion has increased sediment N-loss (831.09 ± 55.16 t N yr−1) but reduced N retention (783.48 ± 37.73 t N yr−1) in mangrove habitats of China within last 40 years (1980–2020). Thus, Spartina alterniflora invasion enhances sediment N-loss while reducing N retention in mangrove wetlands, which may threaten the mangrove buffer function and wetland N pools protection in estuarine and coastal ecosystems.