This manuscript presents an innovative approach to optimize power system protection through strategic relay placement in distribution systems with distributed generation (DG). Traditionally, distribution systems relied on radial configurations, assuming power flow from the grid feeder to downstream networks. However, with the integration of DG technologies, the complexity of relay placement and maintenance increases. The study aims to address protection system issues associated with connecting DGs, such as tripping of production units, blinding of protection, and undesirable islanding. The proposed methodology combines graph theory, energy not supplied (ENS) values, and relay coordination strategies to achieve reliable power system operation. The algorithm is implemented in MATLAB, utilizing data from DigSilent PowerFactory. The key constraints for relay placement, including islanding operation, relay coordination, and load priorities, are considered to minimize the number of power outages and increase overall system reliability. The effectiveness of the algorithm is demonstrated using IEEE 33-bus and 69-bus test systems under different conditions. Results show consistent and reliable relay placement locations, considering DG locations and load priorities. The algorithm's speed, effectiveness, and adaptability to different network topologies make it a promising approach for power system protection planning in distribution systems with distributed generation.