The analysis of the coupled shaft-foundation system (CSFS) of a water turbine generator set is an extremely complicated task owing to its structural complexity and inherent nonlinear broadband dynamic characteristics. Thus, the coupling relationships of the CSFS remain unclear. The broadband vibration of the foundation subsystem has an evident influence on the vibration of the shaft subsystem of the unit; however, for the analysis of the dynamic characteristics of the water turbine generator, an overly complicated foundation can be reasonably simplified. To this end, this paper reports the development of novel energy-equivalent broadband modeling approaches to investigate the nonlinear vibration characteristics of CSFS. First, a relatively traditional integer-order lumped parameter model of the foundation subsystem was constructed, and its parameters were determined by fitting the response spectra obtained from an accurate hydropower house model. To improve the narrow-frequency limitation of the integer-order model and account for random disturbances, a novel fractional-order broadband modeling method is proposed. The advantage of the proposed fractional-order equivalent foundation model is that it can simultaneously reveal the broadband characteristics of complex structures at low frequencies of less than 1 Hz to high frequencies of hundreds of Hz with a single degree of freedom. Finally, coupled dynamic equations for the fractional-order CSFS model were derived and solved by considering the actions of various nonlinear factors and hydraulic pulsation excitation. A comparison of the dynamic responses of the CSFS with field test data demonstrated the effectiveness and practicability of the proposed modeling approaches. The results of the integer- and fractional-order CSFS models provide insights into the abundant dynamic laws and inner mechanisms of the nonlinear coupled dynamic system. Furthermore, they provided a reliable theory for the design and stability analysis of a hydropower unit, considering the varied vibrations of the coupled foundation.