Due to the increasing penetration of intermittent renewable energy and highly stochastic load behavior, it is challenging to effectively assess conservation voltage reduction (CVR) in power distribution systems. This paper proposes a robust time-varying load modeling technique to accurately identify load-to-voltage (LTV) dependence, yielding an improved CVR assessment scheme. In particular, we propose a robust recursive least squares (RLS) approach to estimate time-varying parameters of a ZIP load model at the substation level. Based on the identified load model, we are able to effectively evaluate LTV and analyze CVR factors. We propose a RLS with variable forgetting factors to capture the variations of model parameters under different situations, including continuous and sudden changes of parameters. To further enable RLS to suppress bad or missing measurements, we advocate to use the Huber M-estimator with a convex cost function. Finally, the robust RLS is solved by an iteratively reweighted technique. We demonstrate the effectiveness and the robustness of the proposed method using both simulations and field tests.