Legacy phosphorus (P) accumulated in watersheds from excessive historical P inputs is recognized as an important component of water pollution control and sustainable P management in watersheds worldwide. However, little is known about how watershed P buffering capacity responds to legacy P pressures over time and how long it takes for riverine P concentrations to recover to a target level, especially in developing countries. This study examined long-term (1960–2010) accumulated legacy P stock, P buffering capacity and riverine TP flux dynamics to predict riverine P-reduction recovery times in the Yongan watershed of eastern China. Due to a growing legacy P stock coupled with changes in land use and climate, estimated short- and long-term buffering metrics (i.e., watershed ability to retain current year and historically accumulated surplus P, respectively) decreased by 65% and 36%, respectively, resulting in a 15-fold increase of riverine P flux between 1980 and 2010. An empirical model incorporating accumulated legacy P stock and annual precipitation was developed (R2 = 0.99) and used to estimate a critical legacy P stock of 22.2 ton P km−2 (95% CI 19.4–25.3 ton P km−2) that would prevent exceedance of a target riverine TP concentration of 0.05 mg P L−1. Using an exponential decay model, the recovery time for depleting the estimated legacy P stock in 2010 (29.3 ton P km−2) to the critical level (22.2 ton P km−2) via riverine flux was 456 years (95% CI 353–560 years), 159 years (95% CI 57–262 years) and 318 years (95% CI 238–400 years) under scenarios of a 4% reduction in annual P inputs, total cessation of P inputs, and 4% reduction of annual P inputs with a 10% increase in average annual precipitation, respectively. Given the lower P buffering capacity and lengthening recovery time, strategies to reduce P inputs and utilize soil legacy P for crop production are necessary to effectively control riverine P pollution and conserve global rock P resources. A long-term perspective that incorporates both contemporary and historical information is required for developing sustainable P management strategies to optimize both agronomic and environmental benefits at the watershed scale.