Representational drift—the gradual continuous change of neuronal representations—has been observed across many brain areas. It is unclear whether drift is caused by synaptic plasticity elicited by sensory experience, or by the intrinsic volatility of synapses. Here, using chronic two-photon calcium imaging in primary visual cortex of female mice, we find that the preferred stimulus orientation of individual neurons slowly drifts over the course of weeks. By using cylinder lens goggles to limit visual experience to a narrow range of orientations, we show that the direction of drift, but not its magnitude, is biased by the statistics of visual input. A network model suggests that drift of preferred orientation largely results from synaptic volatility, which under normal visual conditions is counteracted by experience-driven Hebbian mechanisms, stabilizing preferred orientation. Under deprivation conditions these Hebbian mechanisms enable adaptation. Thus, Hebbian synaptic plasticity steers drift to match the statistics of the environment.