According to the dual stream model of auditory language processing, the dorsal stream is responsible for mapping sound to articulation and the ventral stream plays the role of mapping sound to meaning. Most researchers agree that the arcuate fasciculus (AF) is the neuroanatomical correlate of the dorsal steam; however, less is known about what constitutes the ventral one. Nevertheless, two hypotheses exist: one suggests that the segment of the AF that terminates in middle temporal gyrus corresponds to the ventral stream, and the other suggests that it is the extreme capsule that underlies this sound-to-meaning pathway. The goal of this study was to evaluate these two competing hypotheses. We trained participants with a sound-to-word learning paradigm in which they learned to use a foreign phonetic contrast for signaling word meaning. Using diffusion tensor imaging, a brain-imaging tool to investigate white matter connectivity in humans, we found that fractional anisotropy in the left parietal-temporal region positively correlated with the performance in sound-to-word learning. In addition, fiber tracking revealed a ventral pathway, composed of the extreme capsule and the inferior longitudinal fasciculus, that mediated auditory comprehension. Our findings provide converging evidence supporting the importance of the ventral steam, an extreme capsule system, in the frontal-temporal language network. Implications for current models of speech processing are also discussed.