Climate change and human activities are the major factors affecting runoff and sediment load. We analyzed the inter-annual variation trends of the annual rainfall, air temperature, runoff and sediment load in the Xihe River Basin from 1969–2015. Pettitt’s test and the Soil and Water Assessment Tool (SWAT) model were used to detect sudden changes in hydro-meteorological variables and simulate the basin hydrological cycle, respectively. According to the simulation results, we explored spatial distribution of soil erosion in the watershed by utilizing ArcGIS10.0, analyzed the average soil erosion modulus by different types of land use, and quantified the contributions of climate change and human activities to runoff and sediment load in changes. The results showed that: (1) From 1969–2015, both rainfall and air temperature increased, and air temperature increased significantly (p < 0.01) at 0.326 °C/10 a (annual). Runoff and sediment load decreased, and sediment load decreased significantly (p < 0.01) at 1.63 × 105 t/10 a. In 1988, air temperature experienced a sudden increased and sediment load decreased. (2) For runoff, R2 and Nash and Sutcliffe efficiency coefficient (Ens) were 0.92 and 0.91 during the calibration period and 0.90 and 0.87 during the validation period, for sediment load, R2 and Ens were 0.60 and 0.55 during the calibration period and 0.70 and 0.69 during the validation period, meeting the model’s applicability requirements. (3) Soil erosion was worse in the upper basin than other regions, and highest in cultivated land. Climate change exacerbates runoff and sediment load with overall contribution to the total change of −26.54% and −8.8%, respectively. Human activities decreased runoff and sediment load with overall contribution to the total change of 126.54% and 108.8% respectively. The variation of runoff and sediment load in the Xihe River Basin is largely caused by human activities.