Despite bringing several benefits to modern power systems, distributed generation (DG) and microgrids cause a significant impact on the operation of power distribution networks, especially with regard to the protection system. The topological evolution of power distribution networks comprising DG units and the islanding capacity of microgrids have lead to significant changes in the magnitudes of short-circuit currents, consequently affecting the reliability of protection systems in terms of loss of selectivity (false tripping and/or sympathetic tripping), sensitivity (blind protection), and dynamic response (delayed trip). In this new scenario, the adoption of smart protection systems capable of incorporating advanced control and protection mechanisms is of paramount importance for ensuring the automatic reconfiguration and restoration of the power grid when undesirable phenomena like faults occur. Considering this scenario, this work proposes a novel metaheuristic called adaptive fuzzy directional bat algorithm (AFDBA), which can calculate optimal settings for directional overcurrent relays (DOCRs) in any power grid topology without the need for tuning the initial parameters. The technique is applied to three case studies, that is, the 3-bus, 9-bus, and 30-bus test systems. The algorithm performance is compared with that of genetic algorithm (GA), particle swarm optimization (PSO), differential evolution (DE), bat algorithm (BA), and directional bat algorithm (DBA). The results evidence that the proposed approach has high convergence speed, high robustness, and acceptable computational burden, while providing the proper coordination of protection systems.