A low-alloyed Mg-1.2Zn-0.1Ca (wt.%) alloy was fabricated via low-temperature extrusion and annealing at 250 °C for different times (10, 30, and 90 min) to attain heterostructures with different fine-grained fractions, focusing on the effect of heterostructure on the mechanical properties. Partial dynamic recrystallization (RX) occurred during extrusion at 150 °C, and a lamellar structure consisting of fine RX grains and coarse unRX grains was obtained. The subsequent annealing promoted static RX in the as-extruded alloy, leading to an increased fine-grained fraction from 67% to 95%. Meanwhile, the co-segregation of Zn and Ca atoms impeded the migration of grain boundaries, thus achieving a fine grain size of 0.8–1.6 μm. The sample annealed for 10 min with a fine-grained fraction of 73% and an average RX grain size of 0.9 μm exhibited a superior combination of high yield strength (305 MPa) and good ductility (20%). In comparison, an excellent elongation of 30% was achieved in the alloy with a nearly fully-RXed microstructure and an average grain size of 1.6 μm after 90 min annealing, despite a lower yield strength of 228 MPa. In unRX grains, the hard orientation with 〈01–10〉 parallel to the extrusion direction and high-density dislocations made it more difficult to deform compared with the RX grains, thus producing hetero-deformation induced (HDI) strengthening. Besides fine grains and high-density dislocations, HDI strengthening is the key to achieving the superior mechanical properties of the low-alloyed Mg alloy.