In order to match observed properties of the solar cycle, flux-transport dynamo models require the toroidal magnetic flux to be stored in a region of low magnetic diffusivity, typically located at or below the bottom of the convection zone. We infer the turbulent magnetic diffusivity affecting the toroidal field on the basis of empirical data. We consider the time evolution of mean latitude and width of the activity belts of solar cycles 12--23 and their dependence on cycle strength. We interpret the decline phase of the cycles as a diffusion process. The activity level of a given cycle begins to decline when the centers of its equatorward propagating activity belts come within their width (at half maximum) from the equator. This happens earlier for stronger cycles because their activity belts are wider. From that moment on, the activity and the belt width decrease in the same manner for all cycles, independent of their maximum activity level. In terms of diffusive cancellation of opposite-polarity toroidal flux across the equator, we infer the turbulent diffusivity experienced by the toroidal field, wherever it is located, to be in the range 150--450~km$^2$s$^{-1}$. Strong diffusive latitudinal spreading of the toroidal flux underneath the activity belts can be inhibited by an inflow towards the toroidal field bands in the convection zone with a magnitude of several meters per second. The inferred value of the turbulent magnetic diffusivity affecting the toroidal field agrees, to order of magnitude, with estimates based on mixing-length models for the solar convection zone. This is at variance with the requirement of flux-transport dynamo models. The inflows required to keep the toroidal field bands together before they approach the equator are similar to the inflows towards the activity belts observed with local helioseismology.