In continental subduction, rifted margins can be carried to mantle depths (> 90 km) where ultra-high pressure (UHP) metamorphism above coesite stability is attained. Although different exhumation mechanisms for UHP rocks have been discussed, none of them integrate the recent understanding of rifted continental margins. Here, we perform high-resolution thermomechanical numerical experiments to demonstrate that segments of magma-poor rifted margins that reach UHP conditions can efficiently exhume back to shallower levels, while segments of magma-rich rifted margins cannot. This is because the thick layer of rocks with a basaltic composition in magma-rich margins becomes negatively buoyant during metamorphism, preventing their exhumation. This new concept might be pivotal for explaining why exhumed UHP rocks, a key feature of modern-style continental orogens, only appeared and became common late in Earth's history. We suggest that higher mantle potential temperatures and fertility in Earth's early history favored magma-rich rifted margins, making exhumation of UHP crust inefficient. Conditions for magma-poor rifted margins may have arisen during Earth's middle age (1.5–0.8 Ga) due to a colder, more refractory mantle that limited melting and magmatism. We argue that at the end of Neoproterozoic, these colder and positively buoyant magma-poor rifted margins were then subducted and efficiently exhumed to form the large collisional orogens of Gondwana where Earth's oldest coesite-bearing UHP rocks have been unequivocally found. Since then, UHP rocks have become a key and ubiquitous feature of continental orogens.