Objectives: Ketamine exerts rapid antidepressant effects by enhancing neuroplasticity, particularly in the amygdala and hippocampus—regions involved in fear processing and learning. While the role of ketamine’s dissociative effects in its antidepressant response is debated, anxiety experienced during infusion has been negatively correlated with treatment outcomes. Methods: In this single-blind, placebo-controlled study, a subset of 17 healthy volunteers (6 males, 23.12 ± 1.9 years) received intravenously a placebo in the first and 0.5 mg/kg racemic ketamine in the second session. Anxiety-related experiences were assessed by the 5D-ASC score obtained post-infusion, structural magnetic resonance imaging scans were acquired 4 h post-infusion. An anxiety-score was obtained from the 5D-ASC. Relation between post-placebo amygdala volume, hippocampal volume, and its subfields with the anxiety-score were assessed using linear regression models. Results: Results showed a statistically significant negative relation between hippocampal head volume and the anxiety score (β = −0.733, p = 0.006), with trending negative association for each subfield’s head and the score. Conclusion: These findings suggest that anxiety-related experiences during ketamine infusion may be mediated by the hippocampus, with smaller hippocampal volumes leading to more anxiety-related experiences. Thus, hippocampal subfield volumes may be used as a predictor for anxiety-related events during ketamine use and might predict treatment outcome in future approaches.