In the domain of electric power systems, ensuring stable voltage and efficient transmission is crucial. Optimal reactive power dispatch (ORPD) is a vital step toward improving system stability, enhancing economic feasibility, and increasing overall efficiency. Various optimization techniques are employed to minimize real power loss and voltage deviation in the network. This article proposes a hybrid algorithm called Enhanced Jaya and Artificial Ecosystem-Based Optimization (EJAEO) to address the ORPD problem. The algorithm aims to determine the optimal values of variables such as reactive compensation, generators’ voltages, and transformers’ tape ratio. Two single objective functions are tested on standard IEEE 30-Bus and IEEE 57-Bus systems, and the suggested modification of the Artificial Ecosystem-Based Optimization (AEO) improves population effectiveness in achieving optimal solutions. The simulation findings of the EJAEO algorithm demonstrated its superiority over three recent algorithms (AEO), Turbulent Flow of Water-Based Optimization, and Jaya algorithm), as well as certain previously published ORPD methods used for the same problem. In the case of the IEEE 30-Bus system, the proposed EJAEO algorithm achieved the best results in terms of power losses (4.944805) and voltage deviation (0.121196). Similarly, in the IEEE 57-Bus system, our EJAEO algorithm outperformed others, yielding the lowest power losses (23.33052) and voltage deviation (0.579286). The EJAEO algorithm outperforms other algorithms in terms of accuracy, speed of convergence, and robustness. These findings highlight the potential of EJAEO as a promising technique for solving the ORPD problem in power systems.