The network architecture of the human brain contributes in shaping neural activity, influencing cognitive and behavioral processes. The availability of neuroimaging data across the lifespan allows us to monitor how this architecture reorganizes, influenced by processes like learning, adaptation, maturation, and senescence. Changing patterns in brain connectivity can be analyzed with the tools of network science, which can be used to reveal organizational principles such as modular network topology. The identification of network modules is fundamental, as they parse the brain into coherent sub-systems and allow for both functional integration and segregation among different brain areas. In this work we examined the brain’s modular organization by developing an ensemble-based multilayer network approach, allowing us to link changes of structural connectivity patterns to development and aging. We show that modular structure exhibits both linear and nonlinear age-related trends. In the early and late lifespan, communities are more modular, and we track the origins of this high modularity to two different substrates in brain connectivity, linked to the number and the weights of the intra-clusters edges. We also demonstrate that aging leads to a progressive and increasing reconfiguration of modules and a redistribution across hemispheres. Finally, we identify those brain regions that most contribute to network reconfiguration and those that remain more stable across the lifespan.