This article presents a performance evaluation of Prony method vs conventional digital filters in distance relay protection algorithms, and characterizes the error when asynchronous frequency components (subharmonics or interharmonics) are embedded in voltage and current signals during fault period when conventional digital filters of distance relay algorithms (cosine or Fourier filter) are used to estimate the fundamental frequency phasors. First, the signal processing overview of conventional distance relays is presented. Then, the effects of line series compensation in distance relay estimation during the fault period are presented by analyzing two events of recorded data signals in a laboratory test system of a series compensated line and using a mho operation characteristic. Next, Prony method and its considerations for the implementation in distance relay algorithms are described. Then, the performance of Prony method using simulated signals is evaluated. Lastly, an assessment of Prony as a digital filter in distance relay algorithm is performed by analyzing two fault events in a laboratory test system using a quadrilateral operation characteristic, and the results of Prony vs conventional digital filters in distance relay algorithms are presented.