Abstract At several non-volcanic margins serpentinized peridotites occur within a wide continent-ocean transition (COT) and beneath the edge of the thinned continental crust. However, other margins such as the Woodlark Basin appear to have a sharp COT and no reported serpentinites. We investigate the thermal, magmatic and rheological evolution of margins during extension as a function of initial lithospheric structure, rift duration and stretching factor. For cratonic and old orogen models, the entire crust should become brittle at stretching factors c. 3–4. The resultant crust-cutting faults allow water to reach and serpentinize the mantle, leading to the development of serpentinite décollements at the crust-mantle boundary and exhumation of mantle at the COT. Our predictions are consistent with the spatial limit and thickness of serpentinites at the SW Greenland and West Iberia margins, and the Rockall Trough. They also explain the absence of a broad zone of unroofed, serpentinized mantle at the COT of the Woodlark Basin: here the crust was too thick and hot for serpentinites to form before break-up. Larger melt production than in the West Iberia type margins and concentration of the lithospheric strength in the crust leads to synchronous crustal separation and lithospheric failure, yielding a sharp COT.