Environmental problems such as eutrophication caused by excessive nutrient discharge are global challenges. There are complex pollution sources of nitrogen (N) discharge in many river basins worldwide. Knowledge of its pollution sources and their respective load contributions is essential to developing effective N pollution control strategies. N loads from all known anthropogenic pollution sources in the Upper Huai River basin of China were simulated with the process-based SWAT (Soil and Water Assessment Tool) model. The performances of SWAT driven by daily and hourly rainfall inputs were assessed and it was found that the one driven by hourly rainfall outperformed the one driven by daily rainfall in simulating both total nitrogen (TN) and ammonia nitrogen (NH4-N) loads. The hourly SWAT model was hence used to examine the spatiotemporal patterns of TN and NH4-N loads and their source attributions. TN load exhibited significant seasonal variations with the largest in summer and the smallest in spring. Despite its declining proportion of contribution downstream, crop production remained the largest contributor of TN load followed by septic tanks, concentrated animal feedlot operations (CAFOs), municipal sewage treatment plants, industries, and scattered animal feedlot operations (SAFOs). There was much less seasonal variation in NH4-N load. CAFOs remained the largest source of NH4-N load throughout the basin, while contributions from industries and municipal sewage treatment plants were more evident downstream. Our study results suggest the need to shift the focus of N load reduction from “end-of-pipe” sewage treatment to an integrated approach emphasizing stakeholder involvement and source prevention.