The label “deaths of despair” for rising US mortality related to drugs/alcohol/suicide seems to implicate emotional distress as the cause. However, a Durkheimian approach would argue that underlying structural factors shape individuals’ behavior and emotions. Despite a growing literature on deaths of despair, no study has directly compared the effects of distress and structural factors on deaths of despair versus other causes of mortality. Using data from the Midlife in the United States study with approximately 26 years of mortality follow-up, we evaluated whether psychological or economic distress, employment status, and social integration were more strongly associated with drug/alcohol/suicide mortality than with other causes. Cox hazard models, adjusted for potential confounders, showed little evidence that psychological or economic distress were more strongly associated with mortality related to drugs/alcohol/suicide than mortality from other causes. While distress measures were modestly, but significantly associated with these deaths, the associations were similar in magnitude for many other types of mortality. In contrast, detachment from the labor force and lower social integration were both strongly associated with drug/alcohol/suicide mortality, more than for many other types of mortality. Differences in the estimated percentage dying of despair between age 25 and 65 were larger for employment status (2.0% for individuals who were neither employed nor retired versus only 0.6% for currently employed) and for social integration (1.9% for low versus 0.7% for high integration) than for negative affect (1.2% for high versus 0.8% for no negative affect). Most of the association between distress and drug/alcohol/suicide mortality appeared to result from confounding with structural factors and with pre-existing health conditions that may influence both the perception of distress and mortality risk. While deaths of despair result from self-destructive behavior, our results suggest that structural factors may be more important determinants than subjective distress.