Dams and reservoirs have significantly altered river flow dynamics worldwide. Accurately representing reservoir operations in hydrological models is crucial yet challenging. Detailed reservoir operation data is often inaccessible, leading to relying on simplified reservoir operation modules in most hydrological models. To improve the capability of hydrological models to capture flow variability influenced by reservoirs, this study proposes a hybrid hydrological modeling framework, which combines a process-based hydrological model with a machine-learning-based reservoir operation module designed to simulate runoff under reservoir operations. The reservoir operation module employs an ensemble of three machine learning models: random forest, support vector machine, and AutoGluon. These models predict reservoir outflows using precipitation and temperature data as inputs. The Soil and Water Assessment Tool (SWAT) then integrates these outflow predictions to simulate runoff. To evaluate the performance of this hybrid approach, the Xijiang Basin within the Pearl River Basin, China, is used as a case study. The results highlight the superiority of the SWAT model coupled with machine learning-based reservoir operation models compared to alternative modeling approaches. This hybrid model effectively captures peak flows and dry period runoff. The Nash-Sutcliffe Efficiency (NSE) in daily runoff simulations shows substantial improvement, ranging from 0.141 to 0.780, with corresponding enhancements in the coefficient of determination (R2) by 0.098–0.397 when compared to the original reservoir operation modules in SWAT. In comparison to parameterization techniques lacking a dedicated reservoir module, NSE enhancements range from 0.068 to 0.537, and R2 improvements range from 0.027 to 0.139. The proposed hybrid modeling approach effectively characterizes the impact of reservoir operations on river flow dynamics, leading to enhanced accuracy in runoff simulation. These findings offer valuable insights for hydrological forecasting and water resources management in regions influenced by reservoir operations.