Language is bounded to the left hemisphere in the adult brain and the functional lateralization can already be observed early during development. Here we investigate whether this is paralleled by a lateralization of the white matter structural language network. We analyze the strength and microstructural properties of language-related fiber tracts connecting temporal and frontal cortices with a separation of two dorsal tracts, one targeting the posterior Broca’s area (BA44) and one targeting the precentral gyrus (BA6). In a large sample of young children (3–6 years), we demonstrate that, in contrast to the BA6-targeting tract, the microstructural asymmetry of the BA44-targeting fiber tract significantly correlates locally with different aspects of development. While the asymmetry in its anterior segment reflects age, the asymmetry in its posterior segment is associated with the children’s language skills. These findings demonstrate a fine-grained structure-to-function mapping in the lateralized network and go beyond our current view of language-related human brain maturation.