The global energy crisis, particularly in isolated and remote regions, has increased interest in renewable energy sources (RESs) to meet growing energy demands. Integrating RESs with energy storage systems offers a promising solution to mitigate fluctuations and intermittency, but concerns about cost and reliability remain. This study explores the optimal design of various microgrid configurations, combining photovoltaic (PV), wind turbine (WT), battery energy storage system (BESS), and diesel generator (DG) systems for Najran city, Saudi Arabia, via real-world meteorological and load demand data. The Dwarf Mongoose Optimization Algorithm (DMOA), alongside the salp swarm algorithm (SSA) and whale optimization algorithm (WOA), was applied to minimize the levelized cost of energy (LCOE) while improving system reliability. The results demonstrate that the PV/BESS configuration, although cost-effective with an LCOE of 0.038 USD/kWh, fail to meet reliability constraints with a loss of power supply probability (LPSP) of 0.679. In contrast, the PV, WT, BESS, and DG configurations achieved an LPSP of 1.9 × 10^--8% with an LCOE of 0.199 USD/kWh, offering a robust and reliable solution for the region's energy needs. This paper presents a novel application of the DMOA for optimizing hybrid renewable energy systems, demonstrating its effectiveness in achieving a balance between cost and reliability. This strategy provides a viable approach for sustainable energy planning in similar regions facing energy challenges.