Background Previous studies have shown that environment and health can influence drug use trajectories and the effects of substance use disorder (SUD) treatments. We hypothesized that trajectories of drug use-related problems, based on changes in DSM-5 symptoms, would vary by type(s) of drugs used, health factors, and neighborhood characteristics. Methods We assessed mental and physical health, stress, social instability, neighborhood characteristics (disorderliness and home value), and DSM-5 symptom counts at two study visits, 12 months apart, in a community sample (baseline N = 735) in Baltimore, MD. Three prominent categories of drug-use trajectory were identified with K-means cluster analysis of symptom counts: Persistent (4 or more symptoms at both visits or at Visit 2), Improved (decrease from 4 or more symptoms at Visit 1 to 3 or fewer symptoms at Visit 2), and Low-Stable (3 or fewer symptoms at both visits). Baseline health and neighborhood measures were tested as predictors of trajectory in mediation and moderation models. Results Among people with current opioid- and/or stimulant-use, odds of an Improved trajectory were (1) decreased with neighborhood disorder and social instability, or (2) increased with home value and social instability. Odds of a Low-Stable trajectory were decreased by social instability and stress but increased in those who were older or self-identified as white. Conclusions Trajectories of drug use-related problems are influenced by sociodemographic variables, neighborhood factors, and health. Assessing DSM-5 symptom counts as an outcome measure may be valuable in monitoring or predicting long-term trajectories and treatment effectiveness.