Mechanical behavior and microstructural evolution in rolled Mg-3Al-1Zn-0.5Mn (AZ31B) magnesium alloy under in-plane shear (IPS) and through-thickness shear (TTS) were investigated. The newly designed butterfly-shaped specimens were sheared along the rolling direction, and a full field strain analysis was performed by digital image correlation (DIC). The shear stress-strain curves show that the shear strength and strain hardening rate are significantly different between the IPS and TTS. The electron backscatter diffraction (EBSD) observation shows that extension twinning is the dominant twinning mechanism under shear deformations. Moreover, a certain volume of {10-12}-{10-12} double twins appears throughout the deformation process in the TTS, while a few compression twins and {10-11}-{10-12} double twins occur at the early stage in the IPS. The results suggest that crystal orientation plays an important role in determining the twin types and twin volume fraction in the shear deformations. The visco-plastic self-consistent (VPSC) model is also applied to predict the activity of slip/twinning under shear deformations. The VPSC results confirm that the mechanical discrepancy at early shear deformation is mainly due to the activation of extension twin and prismatic slip.