Diffusion magnetic resonance imaging studies often investigate white matter (WM) microstructural degeneration in aging by probing WM regions that exhibit negative age associations of fractional anisotropy (FA). However, WM regions in which FA is unassociated with age are not necessarily “spared” in aging. Besides the confound of inter-participant heterogeneity, FA conflates all intravoxel fiber populations and does not allow the detection of individual fiber-specific age associations. In this study of 541 healthy adults aged 36–100 years, we use fixel-based analysis to investigate age associations among each “fixel” within a voxel, representing individual fiber populations. We find age associations of fixel-based measures that indicate age-related differences in individual fiber populations amid complex fiber architectures. Different crossing fiber populations exhibit different slopes of age associations. Our findings may provide evidence of selective degeneration of intravoxel WM fibers in aging, which does not necessarily manifest as a change in FA and therefore escapes notice if conventional voxel-based analyses are relied upon alone.