Effects of neutron irradiation on the microstructure and mechanical properties of CMRR (China Mianyang Research Reactor) structural material pure aluminum and 6061 alloy were studied by scanning/transmission electron microscopy (S/TEM) and tensile tests. The fast neutron (E > 0.1 MeV) fluences of 1.09–2.89 × 1022 n•cm−2 produced damage doses of 1.6–4.2 displacements per atom (dpa). Voids and faulted 1/3<111> dislocation loops were observed in both pure aluminum and 6061 alloy. In particular, voids tended to preferentially aggregate on coarse Cu-rich and fine Mg2Si precipitates which contributes to the higher irradiation tolerance in terms of void formation for 6061 alloy, suggesting that the uniformly distributed precipitates can act as trapping sites for defect clusters during irradiation and thus promote recombination. The size and density of Mg2Si precipitates showed a slight increase and Cu-rich precipitates emerged in 6061 alloy after neutron irradiation. Moreover, radiation-induced segregation was also detected along the grain boundaries. After tensile testing, the pure aluminum showed a lower yield strength and larger uniform elongation than 6061 alloy, where a moderate uniform elongation was still reserved after irradiation to 4.2dpa. The severely reduced ductility in 6061 alloy is supposed to be caused by the transmutation product silicon which contributes to the increase of dispersed multiple precipitates after irradiation. Our results indicate that precipitates especially Mg2Si play a decisive role in void suppression, but are not suitable in diminishing the radiation hardening.