Androgen signaling is central to prostate cancer and men's health. Prior data indicates that increasing body fat is unfavorable in the localized setting yet associated with favorable outcomes in men with metastatic disease. Understanding the biological links between adiposity and prostate cancer may optimize the therapeutic index with ASI. We hypothesized that host adiposity and androgen synthesis are linked to the efficacy and toxicity of ASI for men with metastatic castration-resistant prostate cancer (mCRPC). A post-hoc analysis was done of NCT02703623 where men with mCRPC (n = 186) were treated for 8 weeks with abiraterone acetate, prednisone, and apalutamide (AAPA), and a satisfactory response was defined as a PSA decline >50%. Body composition was measured on baseline CT scans. Germline DNA WES was performed with a focus on variants in steroidogenic genes. Adipokine levels were measured in pre-treatment plasma. Germline polymorphisms in 3 genes involved in androgen synthesis (AKR1C3 rs12529, CYP17A1 rs6162, SRD5A2 rs523349) were associated with differences in body composition at baseline on ADT alone (prior to receipt of AAPA). Elevated subcutaneous adipose tissue index (SATi, p = 0.02), visceral adipose tissue index (VATi, p = 0.03), and BMI (p = 0.04) were associated with satisfactory response to AAPA. Leptin had positive correlation with VATi (r = 0.47) and SATi (r = 0.48). Inherited polymorphisms in androgen synthesis correlated with differences in body composition after exposure to ADT and warrant further investigation as candidate markers for body composition toxicity. Elevated subcutaneous and visceral adiposity were associated with improved response to ASI.