Ammonia is a major inhibitor in anaerobic digestion of nitrogen-rich organic wastes. In this study, integrated genome-centric metagenomic and metaproteomic analyses were used to identify the key microorganisms and metabolic links causing instability by characterizing the process performance, microbial community, and metabolic responses of key microorganisms during endogenous ammonia accumulation. The identification of 89 metagenome-assembled genomes and analysis of their abundance profile in different operational phases permitted the identification of key taxa (Firmicutes and Proteobacteria) causing poor performance. Metabolic reconstruction indicated that the key taxa had the genetic potential to participate in the metabolism of C2C5 volatile fatty acids (VFAs). Further investigation suggested that during Phase I, the total ammonia nitrogen (TAN) level was maintained below 2000 mg N/L, and the reactor showed a high methane yield (478.30 ± 33.35 mL/g VS) and low VFAs concentration. When the TAN accumulated to > 2000 mg N/L, acid accumulation, mainly of acetate, began to occur, and the methane yield gradually decreased to 330.44 mL/g VS (Phase II). During this phase, the VFA degradation functions of the community were mainly mediated by Firmicutes. Approximately 61.54% of significant differentially expressed proteins (DEPs) related to acetate metabolism in Firmicutes were down-regulated, which led to an increase in acetate concentration to 4897.91 ± 1558.96 mg/L. However, the reactor performance showed spontaneous recovery without any interference (Phase III), during which Firmicutes gradually adapted to the high ammonia conditions. Approximately 75% of the significant DEPs related to acetate metabolism of Proteobacteria were also up-regulated in Phase III compared with Phase II; thus, VFA-related metabolic functions of the community were enhanced, which resulted in a decrease in the total VFA concentration to 195.39 mg/L. When the TAN increased above 4000 mg N/L, the system gradually showed acid accumulation dominated by propionate, accompanied by a second decrease in methane yield (Phase IV). During this phase, the number of up-regulated and down-regulated proteins related to acetate metabolism of Firmicutes and butyrate/valerate metabolism of Proteobacteria was comparable with that of Phase III, indicating that the metabolic functions related to acetate, butyrate, and valerate of the microbial community were not significantly affected. However, for propionate metabolism, the expression activity of fumarate hydratase from Firmicutes and Proteobacteria was severely inhibited by ammonia, as shown by down-regulation ratios of 63.64% and 85.71%, respectively. No protein with the same function that was not inhibited by ammonia could be detected, and the fumarate degradation function of the microbial community was severely damaged, leading to blocked propionate metabolism and irreversible deterioration of reactor performance. This study has provided a new perspective on the microecological mechanisms of ammonia inhibition.