Symptoms of irritability, anxiety, panic, and insomnia are common in patients with depression, and their worsening after antidepressant treatment initiation is associated with poorer long-term outcomes. The Concise Associated Symptom Tracking (CAST) scale was developed to measure these symptoms in adults with major depressive disorder (MDD). Here, we evaluate the psychometric properties of CAST in an ongoing community-based observational study involving children, adolescents, and young adults. Individuals from the ongoing Texas Youth Depression and Suicide Research Network (TX-YDSRN; N = 952) with CAST data available were included. Fit statistics [Goodness of Fit Index (GFI), Comparative Fit Index (CFI), and Root Mean Square Error of Approximation (RMSEA)] from confirmatory factor analyses were used to evaluate the five- and four-domain structure of CAST. Item response theory (IRT) analyses were also used. Individuals were grouped based on age (in years) as youths (8–17) and young adults (18–20). Correlations with other clinical measures were used to inform construct validity. Four-domain (irritability, anxiety, panic, and insomnia) 12-item structure of CAST (CAST-12) was optimal for youths (N = 709, GFI = 0.906, CFI = 0.919, RMSEA = 0.095) and young adults (N = 243, GFI = 0.921, CFI = 0.938, RMSEA = 0.0797) with Cronbach's alpha of 0.87 and 0.88, respectively. Slope of each item exceeded 1.0 on IRT analyses suggesting adequate discrimination for each item. Scores on irritability, anxiety, panic, and insomnia were significantly correlated with similar items on other scales. Together these findings suggest that CAST-12 is a valid self-report measure of irritability, anxiety, insomnia, and panic in youths and young adults.