Two key factors govern how bilingual speakers neurally maintain two languages: the speakers’ second language age of acquisition (AoA) and their subsequent proficiency. However, the relative roles of these two factors have been difficult to disentangle given that the two can be closely correlated, and most prior studies have examined the two factors in isolation. Here, we combine functional magnetic resonance imaging with diffusion tensor imaging to identify specific brain areas that are independently modulated by AoA and proficiency in second language speakers. First-language Mandarin Chinese speakers who are second language speakers of English were scanned as they performed a picture-word matching task in either language. In the same session we also acquired diffusion-weighted scans to assess white matter microstructure, along with behavioural measures of language proficiency prior to entering the scanner. Results reveal gray- and white-matter networks involving both the left and right hemisphere that independently vary as a function of a second-language speaker's AoA and proficiency, focused on the superior temporal gyrus, middle and inferior frontal gyrus, parahippocampal gyrus, and the basal ganglia. These results indicate that proficiency and AoA explain separate functional and structural networks in the bilingual brain, which we interpret as suggesting distinct types of plasticity for age-dependent effects (i.e., AoA) versus experience and/or predisposition (i.e., proficiency).