In this paper, quasi-in situ experiments were carried out on rolled AZ31 magnesium alloy sheets to track the recrystallization behavior of the rolled microstructure during the heat treatment process and the plastic deformation behavior during the stretching process. The as-rolled microstructures are classified into five characteristics and their plastic deformation behaviors are described. The research shows that annealing recrystallization leads to grain reorganization, resulting in the diversity of grain orientation, and it is easier to activate basal slip. Recrystallization preferentially nucleates in the regions with high stress, while it is difficult for recrystallization to occur in regions with low stress, which leads to the uneven distribution of the as-rolled structure of magnesium alloys. Slip can be better transmitted between small grains, while deformation between large and small grains is difficult to transmit, which can easily lead to the generation of ledges. Incomplete recrystallization is more likely to accumulate dislocations than complete recrystallization, and ledges are formed in the early stage of deformation. Microcracks are more likely to occur between strain-incompatible grains. It is of great significance to promote the application of rolled AZ31 magnesium alloys for the development of heat treatment and subsequent plastic working of rolled magnesium alloys.