ABSTRACT Detailed study of the solar magnetic field is crucial to understand its generation, transport, and reversals. The timing of the reversals may have implications on space weather and thus identification of the temporal behaviour of the critical surges that lead to the polar field reversals is important. We analyse the evolution of solar activity and magnetic flux transport in Cycles 21–24. We identify critical surges of remnant flux that reach the Sun’s poles and lead to the polar field reversals. We reexamine the polar field build-up and reversals in their causal relation to the Sun’s low-latitude activity. We further identify the major remnant flux surges and their sources in the time-latitude aspect. We find that special characteristics of individual 11-yr cycles are generally determined by the spatiotemporal organization of emergent magnetic flux and its unusual properties. We find a complicated restructuring of high-latitude magnetic fields in Cycle 21. The global rearrangements of solar magnetic fields were caused by surges of trailing and leading polarities that occurred near the activity maximum. The decay of non-Joy and anti-Hale active regions resulted in the remnant flux surges that disturbed the usual order in magnetic flux transport. We finally show that the leading-polarity surges during cycle minima sometimes link the following cycle and a collective effect of these surges may lead to secular changes in the solar activity. The magnetic field from a Babcock–Leighton dynamo model generally agrees with these observations.