This paper presents a novel load frequency control (LFC) model for an interconnected thermal two-area power system in the presence of wind turbine generation and redox flow battery (RFB). The study model includes frequency and voltage excitation loops with needed interactions between them along with the power system stabilizer. A two-degree of freedom (2DOF)-based controller called 2DOF-Hybrid controller is developed as secondary controller in automatic generation control (AGC) to adjust the power outputs of generator and RFB. Also, the dynamic performance of the proposed controller in the RFB loop is evaluated. The Hybrid controller comprises a fractional-order proportional-integral-derivative (FOPID) controller and a tilt-integral-derivative (TID) controller. In order to obtain accurate and realistic results, the outputs of thermal power plants are restricted by considering the limitations of the governor dead-band and generation rate constraint. Since the controller performance depends on its parameters, these parameters are optimized using a modified sine–cosine algorithm (MSCA). The dynamic performance of the proposed 2DOF-Hybrid controller as secondary controller of the AGC loop is compared with integral-double-derivative (IDD), integral-tilt-derivative, proportion-integral–derivative (PID)-DD, 2DOF-PID, 2DOF-TID, and 2DOF-FOPID ones under different scenarios. In addition, the superiority of the MSCA is compared with benchmark metaheuristic methods including an SCA, a genetic algorithm, a particle swarm optimization, and a differential evolution. The sensitivity analysis is also carried out to show the robustness of the proposed controller versus the changes of the parameters. The simulation studies on two-area and New England 39-bus power systems are carried out to examine the advantage of the presented LFC scheme. A range of power system signals such as frequencies of areas, terminal voltages, and tie-line power flow is demonstrated to compare the controllers. The results disclose that the proposed LFC scheme provides better dynamic performance compared to other ones. Moreover, RFB modeling based on the proposed controller is superior to conventional RFB modeling in reducing the amplitude of the oscillations.