The collapse of passive margins has been proposed as a possible mechanism for the spontaneous initiation of subduction. In order for a new trench to form at the junction between oceanic and continental plates, the cold and stiff oceanic lithosphere must be weakened sufficiently to deform at tectonic rates. Such rates are especially hard to attain in the cold ductile portion of the lithosphere, at which the mantle lithosphere reaches peak strength. The amount of weakening required for the lithosphere to deform in this tectonic setting is dictated by the available stress. Stress in a cooling passive margin increases with time (e.g., due to ridge push), and is augmented by stresses present in the lithosphere at the onset of rifting (e.g., due to drag from underlying mantle flow). Increasing stress has the potential to weaken the ductile portion of the lithosphere by dislocation creep, or by decreasing grain size in conjunction with a grain-size sensitive rheology like diffusion creep. While the increasing stress acts to weaken the lithosphere, the decreasing temperature acts to stiffen it, and the dominance of one effect or the other determines whether the margin might weaken and collapse. Here, we present a model of the thermal and mechanical evolution of a passive margin, wherein we predict formation of a weak shear zone that spans a significant depth-range of the ductile portion of the lithosphere. Stiffening due to cooling is offset by weakening due to grain size reduction, driven by the combination of imposed stresses and grain damage. Weakening via grain damage is modest when ridge push is the only source of stress in the lithosphere, making the collapse of a passive margin unlikely in this scenario. However, adding even a small stress-contribution from mantle drag results in damage and weakening of a significantly larger portion of the lithosphere. We posit that rapid grain size reduction in the ductile portion of the lithosphere can enable, or at least significantly facilitate, the collapse of a passive margin and initiate a new subduction zone. We use this model to estimate the conditions for passive margin collapse for modern and ancient Earth, as well as for Venus.