Al-Mg alloys are widely used as important engineering structural materials in aerospace engineering, transportation systems, and structural constructions due to their low density, high specific strength, corrosion resistance, welding capability, fatigue strength, and cost-effectiveness. However, the conventional Al-Mg alloys can no longer fully satisfy the demands of practical production due to difficulties caused by many defects. The high strength of Al-Mg alloys as non-heat treatment precipitation-strengthened alloys is achieved primarily by solid solution strengthening along with work hardening rather than precipitation strengthening. Therefore, severe plastic deformation (SPD) techniques can be often used to produce ultrafine-grained structures to fabricate ultra-high strength aluminum alloys. However, this approach often achieves the strengthening of material at the cost of reduced ductility. This paper comprehensively summarizes the various approaches of ultrafine/nanocrystalline materials for enhancing their plasticity, elaborates on the creation of a bimodal microstructure within the alloy, and discusses the formation of a nanotwin microstructure within the alloy and the incorporation of dispersed nanoparticles. The mechanisms underlying both the strengthening and toughening during large plastic deformation in aluminum alloys are summarized, and the future research direction of high-performance ultrafine crystalline and nanocrystalline Al-Mg aluminum alloys is prospected.