This article proposes a novel approach for the optimal coordination of adaptive directional overcurrent relay (ADOCR) for modernized active distribution system (ADS) with microgrid (MG) functions. The goal is to keep the coordinating circumstances while reducing the protective relay operation time, as much as possible. With the proposed round-off mixed-integer particle swarm optimization (ROMI-PSO), the setting parameters are not limited to only relay time multiplier setting (TMS) and pickup current setting (PS) but can include the discrete parameter as relay characteristic curve setting (CS). In the proposed framework, The CS can be both IEC and IEEE standard curves in the proposed framework. Four buses radial system and IEC-benchmark MG system were used in a number of simulation experiments. Comparing the suggested methodology to other current methods revealed that it is more effective at determining the minimum relay operating times and ensuring the proper operation of protections under various fault allocations and operating circumstances.