The wide application of power electronic devices brings an increasing amount of undesired harmonic and interharmonic tones, and accurate harmonic phasor estimation under a complex signal input is an important task for smart grid applications. In this paper, an optimization of least-square dynamic harmonic phasor estimators, considering multi-interference and harmonic frequency variance, is proposed. A comprehensive error index (CEI) composed of the fundamental-leakage-led harmonic amplitude estimation error, harmonic mutual interference, out-of-band interference, and harmonic frequency deviation is employed. The largest CEI part of least-square algorithms using three different signal decomposition models is analyzed for the first time, and variables to reduce this error component are then introduced using singular value decomposition. With the CEI and defined variables, a minimum-error estimation of harmonic phasors under various interference and harmonic frequency change is discussed. Numerical tests are performed, and the test results show that after the proposed optimization is applied to least-square algorithms, the harmonic phasor estimation errors are considerably reduced, especially for low-order harmonics. We also show the possibility of choosing desired optimal phasor filter design by balancing the measurement accuracy and data latency. For example, when the window length is set to three nominal cycles, the proposed optimization can yield both good accuracy and fast measurement speed for estimating harmonic phasors under multi-interference and harmonic frequency variance.