Mg-4.1Al-0.6Mn-0.5Zn-0.4Ca alloy with high strength was prepared through hot extrusion at 330 °C followed by room temperature rotary swaging. The changes of microstructure and mechanical behaviors of extruded and swaged alloys after undergoing different rotary waging passes were investigated. The results reveal significant microstructural refinement, fragmentation of the Al8Mn5 phases, and precipitation of the Al2Ca phases after rotary swaging. With the increase of rotary swaging passes, a noticeable rise in the fraction of LAGBs and a more uniform distribution and refinement of second phases occurred. The texture with <10–10> parallel to processing direction was formed after rotary swaging. The rotary swaging largely improves the alloy's strength, with the ultimate tensile strength and yield strength increasing from 298 MPa to 198 MPa for the as-extruded alloy to 372 MPa and 359 MPa for the as-swaged alloy after 13 passes, respectively. The improvement of mechanical properties is a result of grain boundary strengthening, dislocation strengthening, and the strengthening of second phases. This result may provide valuable insights and guidance in the design of high-strength magnesium alloys as well as their plastic processing methods.