Rotation evolution of late-type stars is dominated by magnetic braking and the underlying factors that control this angular momentum loss are important for the study of stellar spin-down. In this work, we study angular momentum loss as a function of two different aspects of magnetic activity using a calibrated Alfv\'en wave-driven magnetohydrodynamic wind model: the strengths of magnetic spots and their distribution in latitude. By driving the model using solar and modified solar surface magnetograms, we show that the topology of the field arising from the net interaction of both small-scale and large-scale field is important for spin-down rates and that angular momentum loss is not a simple function of large scale magnetic field strength. We find that changing the latitude of magnetic spots can modify mass and angular momentum loss rates by a factor of two. The general effect that causes these differences is the closing down of large-scale open field at mid- and high-latitudes by the addition of the small-scale field. These effects might give rise to modulation of mass and angular momentum loss through stellar cycles, and present a problem for ab initio attempts to predict stellar spin-down based on wind models. For all the magnetogram cases considered here, from dipoles to various spotted distributions, we find that angular momentum loss is dominated by the mass loss at mid-latitudes. The spin-down torque applied by magnetized winds therefore acts at specific latitudes and is not evenly distributed over the stellar surface, though this aspect is unlikely to be important for understanding spin-down and surface flows on stars.