We estimate the relaxation time $\ensuremath{\tau}$ of an average electron velocity relative to a dilute array of vortex cores in a rotating, $s$-wave-paired neutron superfluid. At low temperatures, $\ensuremath{\tau}$ is found to vary exponentially with $\frac{{\ensuremath{\Delta}}^{2}}{{\ensuremath{\epsilon}}_{F}}$, the energy scale of vortex core excitations, where $\ensuremath{\Delta}$ is the gap parameter and ${\ensuremath{\epsilon}}_{F}$ is the neutron Fermi energy. For reasonable choices of $\ensuremath{\Delta}$ and ${\ensuremath{\epsilon}}_{F}$, we find values of $\ensuremath{\tau}$ which include the values of a year and of several days observed, respectively, in the post-speedup relaxation of the Vela and Crab pulsars' slowdown rates.