Crop straw return and organic manure application are effective and environmentally friendly methods used by farmers to improve soil structure, fertility and agricultural productivity. Sweet potato is a typical “potassium (K)-favoring” food crop and widely grown throughout the world. However, the regulation effects of long-term organic fertilization on soil aggregates-associated K stocks and K uptake by sweet potato are less understood. Herein, a long-term fertilization experiment since 2007 including chemical fertilizer (NPK), NPK combined with commercial manure (NPK+CM), NPK combined with pig manure (NPK+PM), and NPK combined with rice straw (NPK+RS) treatments under peanut-sweet potato crop rotation was employed. Long-term fertilizer application contributed to higher yield, yield stability, and K uptake of sweet potato. NPK+PM and NPK+RS treatments both showed the best promoting effect on plant growth. NPK+PM and NPK+RS treatments showed remarkable increasement of aggregate-associated exchangeable potassium (EK) and non-exchangeable potassium (NEK) stocks, particularly for small-macroaggregates (0.25–2 mm) and large-macroaggregates (> 2 mm). Total potassium (TK), EK and NEK stocks in large and small-macroaggregates displayed positive effects on K uptake and yield for sweet potato. Pearson's correlation and random forest model analysis revealed that EK and NEK stock in large-macroaggregate were the most key factors for K uptake and yield of sweet potato under long-term fertilization. Overall, our study highlights the prominent advantage of long-term organic fertilization.