This paper proposes a robust design of the time-varying internal model principle-based control (TV-IMPC) for tracking sophisticated references generated by linear time-varying (LTV) autonomous systems. The existing TV-IMPC design usually requires a complete knowledge of the plant I/O (input/output) model, leading to the lack of structural robustness. To tackle this issue, we, in this paper, design a gray-box extended state observer (ESO) to estimate and compensate unknown model uncertainties and external disturbances. By means of the ESO feedback, the plant model is kept as nominal, and hence the structural robustness is achieved for the time-varying internal model. It is shown that the proposed design has bounded ESO estimation errors, which can be further adjusted by modifying the corresponding control gains. To stabilize the ESO-based TV-IMPC, a time-varying stabilizer is developed by employing Linear Matrix Inequalities (LMIs). Extensive simulation and experimental studies are conducted on a direct-drive servo stage to validate the proposed robust TV-IMPC with ultra-precision tracking performance (∼60nm RMSE out of ±80mm stroke).