Previous magnetic resonance imaging (MRI) research suggests that aging is associated with a decrease in the functional interconnections within and between groups of locally organized brain regions (modules). Further, this age-related decrease in the segregation of modules appears to be more pronounced for a task, relative to a resting state, reflecting the integration of functional modules and attentional allocation necessary to support task performance. Here, using graph-theoretical analyses, we investigated age-related differences in a whole-brain measure of module connectivity, system segregation, for 68 healthy, community-dwelling individuals 18-78years of age. We obtained resting-state, task-related (visual search), and structural (diffusion-weighted) MRI data. Using a parcellation of modules derived from the participants' resting-state functional MRI data, we demonstrated that the decrease in system segregation from rest to task (i.e., reconfiguration) increased with age, suggesting an age-related increase in the integration of modules required by the attentional demands of visual search. Structural system segregation increased with age, reflecting weaker connectivity both within and between modules. Functional and structural system segregation had qualitatively different influences on age-related decline in visual search performance. Functional system segregation (and reconfiguration) influenced age-related decline in the rate of visual evidence accumulation (drift rate), whereas structural system segregation contributed to age-related slowing of encoding and response processes (nondecision time). The age-related differences in the functional system segregation measures, however, were relatively independent of those associated with structural connectivity.