The nonlinear system of equations (NSE) is the basis of many engineering and scientific models. However, these models must have accurate solutions to be successful. Several optimization algorithms, including Butterfly Optimization Algorithm (BOA), can be applied to solving NSE problems. However, optimization algorithms lack accurate solutions due to their limitations, including trapping at local optima and divergence problems. In this work, a novel hybrid approach that is called Jarratt Butterfly Optimization Algorithm (JBOA) is presented to solve NSE. The proposed JBOA is developed by combining Jarratt’s iterative method and the BOA. Combining the two methods has enhanced BOA’s search mechanism and convergence speed. It also overcomes Jarratt’s limitations, including selecting initial values, trapping at local optima, and divergence problems. This results in solving NSE in more accurate solutions and fewer iterations. JBOA’s efficiency was evaluated based on eight benchmark systems, two of which represent real-world applications. Further, JBOA was compared to several optimization algorithms, including the original BOA algorithm, Particle Swarm Optimization (PSO), Harris Hawk Optimization (HHO), Equilibrium Optimization (EO), and Ant Lion Optimizer (ALO). Furthermore, the performance of JBOA and Jarratt’s method has been compared. Finally, the experimental results support the superiority of JBOA as it significantly outperformed all other compared optimization algorithms in all benchmark systems in terms of best solutions, average fitness values, stability, and convergence speed.