In this paper, we proposed an enhanced reptile search algorithm (RSA) for global optimization and selected optimal thresholding values for multilevel image segmentation. RSA is a recent metaheuristic optimization algorithm depending on the hunting behavior of crocodiles. RSA is inclined to inadequate diversity, local optima, and unbalanced exploitation abilities as other metaheuristic algorithms. The RUNge Kutta optimizer (RUN) is a novel metaheuristic algorithm that has demonstrated effectiveness in solving real-world optimization problems. The enhanced solution quality (ESQ) in RUN utilizes the thus-far best solution to promote the quality of solutions, improve the convergence speed, and effectively balance the exploration and exploitation steps. Also, the Scale factor (SF) has a randomized adaptation nature, which helps RUN in further improving the exploration and exploitation steps. This parameter ensures a smooth transition from exploration to exploitation. In order to mitigate the drawbacks of the RSA algorithm, this paper proposed a modified RSA (mRSA), which combines the RSA algorithm with the RUN. The ESQ mechanism and the scale factor boost the original RSA’s performance, enhance convergence speed, bypass local optimum, and enhance the balance between exploitation and exploration. The validity of mRSA was verified using two experimental sequences. First, we applied mRSA to CEC’2020 benchmark functions of various types and dimensions, showing that mRSA has more robust search capabilities than the original RSA and popular counterpart algorithms concerning statistical, convergence, and diversity measurements. The second experiment evaluated mRSA for a real-world application to solve magnetic resonance imaging (MRI) brain image segmentation. Overall experimental results confirm that the mRSA has a strong optimization ability. Also, mRSA method is a more successful multilevel thresholding segmentation and outperforms comparison methods according to different performance measures.