The relatively high fluxes of the Galactic ultraluminous X-ray pulsar Swift J0243 allow a detailed study of its spin-down regime in the quiescent state for the first time. After the 2017 giant outburst, its spin frequency shows a linearly decreasing trend with some variations due to minor outbursts. The linear spin-down rate is ∼ −1.9 × 10−12 Hz s−1 during the period of lowest luminosity, from which one can infer a dipole field of ∼1.75 × 1013 G. The relation during the spin-down regime is complex, and is close to zero when the luminosity reaches both the high end (L 38 ∼ 0.3) and the lowest value (L 38 ∼ 0.03). The luminosity of zero torque is different for the giant outburst and other minor outbursts. This is likely due to different accretion flows for different types of outburst, as evidenced by the differences in the spectra and pulse profiles at a similar luminosity for different types of outburst (giant or not). The pulse profile changes from double peaks in the spin-up state to a single broad peak in the low spin-down regime, indicating the emission beam/region is larger in the low spin-down regime. These results show that accretion is still ongoing in the low spin-down regime, for which the neutron star is supposed to be in a propeller state.