Poststroke aphasia (PSA) recovery shows high variability across individuals and at different time points. Although diffusion biomarkers from the ventral and dorsal streams have demonstrated strong predictive power for language outcomes, it is still unclear how these biomarkers relate to the various stages of PSA recovery. In this study, we aim to compare diffusion metrics and language measures as predictors of language recovery in a longitudinal cohort of participants with PSA. Participants were recruited at a stroke unit at the emergency room, and underwent diffusion magnetic resonance imaging scanning and language assessment within 3 days (acute phase) after stroke, with behavioral follow-ups at subacute (10±3 days) and chronic phases (>6 months). We conducted regression analyses on language performance (cross-sectional), Δscores between all time points (acute-subacute, subacute-chronic, acute-chronic), and relative Δscores between all time points (Δscore/language baseline score), with acute diffusion metrics from language-related white matter tracts, lesion size, language baseline scores, and demographic data as predictors. Thirty-nine participants presenting PSA were recruited, and 24 participants (mean age, 73 years; 8 women) completed the 3-time point assessment in total. The best prediction model of performance scores used axial diffusivity from the left arcuate fasciculus in both the subacute (R2=0.785) and chronic stages (R2=0.626). Moreover, the prediction of ∆scores depended on axial diffusivity from the left inferior frontal-occipital fasciculus in the subacute stage (R2=0.5) and depended additionally on axial diffusivity from the right inferior frontal-occipital fasciculus in the chronic stage (R2=0.68). The prediction of mediation analyses showed that the lesion load of the left arcuate fasciculus mediated the relationship between axial diffusivity from the left arcuate fasciculus and chronic language performance. Language performance at subacute and chronic time points could be predicted by the integrity of the left arcuate fasciculus, whereas Δscores in the subacute and chronic phases depended on the left inferior frontal-occipital fasciculus, showing a dissociation of the white matter pathways about language outcomes. These results suggest a functional differentiation of the dual-stream components in PSA recovery.