Detection and discrimination of a High Impedance Fault (HIF) is difficult from other noteworthy switching events because of the fault path resistance and current signature. The nonlinear current variation poses several protection challenges to existing methods. To mitigate this issue, a Savitzky-Golay Filter (SGF) integrated with a Matrix Pencil Method (MPM) is used to develop a Transient Detection Index (TDI). The SGF-MPM offers advanced filtering to the noisy signal. The extracted SGF is then reconstructed using MPM and an error is estimated which contains only fault generated components. The Teager energy of the error is estimated and compared with an adaptive threshold for secure detection of HIF. The index is verified for several HIFs and non-HIFs using Aalborg test feeder, and modified IEEE 30 bus test systems. The impacts of noise, harmonics, unbalanced loading of lines, voltage sag, system grounding configuration, distribution operating mode, and different switching events on the performance of the proposed method are verified. A comparison with other existing methods is done. From the detailed analysis, it is found that the percentage of accuracy and dependability for the proposed adaptive HIF detection method are obtained as 98.6% and 98.1% respectively.