AbstractThe phenomenon of imbalance is a variation of voltage and current waveforms from a perfect sinusoidal waveform, which is considered a crucial challenge in power systems. Uneven loads and disturbances result in three‐phase voltage/current unbalance that leads to a temperature rise and low output curves at synchronous machines. Thus, the unbalance/fault must be recognized and avoided. This article suggests a methodology to detect and evaluate three‐phase voltage and current unbalance/fault using a coherence algorithm. The suggested scheme develops integrated protection functions that can be divided into three modules: voltage unbalance, current unbalance, and power factor unbalance. Therefore, it can be useful in mitigation techniques applied to compensate the voltage unbalance effects. The present protection introduces a new proposal of closed‐tripping curves based on the values range of nine coherence factors. To investigate the protection performance, a segment of power grid with real parameters’ information is simulated utilizing ATP platform. Several fault types, such as shunt and series faults, are conducted on the simulated system to analyze the results and monitor the system status using the coherence statistic processed in MATLAB software. The results show that the approach is smart, swift, precise, reliable and stable. Moreover, the algorithm can be practically implemented, and represents a base for digital unbalance/fault detectors and relays because its performance is superior in detecting Power Quality Disturbances (PQDs).