The AZ31 magnesium alloy with different grain sizes was prepared by adjusting the heat treatment process, and then the tension and compression tests were carried out to explore the effect of grain sizes and loading conditions on the microstructures and properties of the magnesium alloy. The Hall-Petch slope presented dramatic anisotropy under tension (181 Mpa· μm1∕2) and compression (115 Mpa· μm1∕2), the geometric compatibility factor (m′) and the difference in activation stress between the initiated deformation modes of two neighboring grains (σd) were calculated to analyze the anisotropy quantitatively. In compression, it was found that the elongation increased with further grain coarsening, which was attributed to the fact that more twins were initiated in the coarse grains and the preferred initiation of twinning to inhibit competitive crack initiation.