Plastic instability caused by the Portevin-Le Chatelier (PLC) effect remains a challenge for Al–Mg alloy sheets during shape-forming processes, as it causes unsightly stretcher-strain markings on the surface of final products. In this study, the PLC behavior in annealed AlMg and AlMgScZr alloy sheets with varying Al3(Sc,Zr) dispersoid distributions and grain structures were investigated by combining tensile tests, microstructural characterization, and DIC analysis. The results show that the presence of Al3(Sc,Zr) dispersoids can inhibit the PLC effect, which is manifested by a decrease in serration amplitude, PLC band velocity, and an increase in critical strain. The inhibition effect is shown to be highly related to the distribution of Al3(Sc,Zr) dispersoids and resultant grain structures. The alloy with the densest dispersoid distribution and finest subgrain structure showed the slightest PLC effect. The reduction in serration amplitude and PLC band velocity is ascribed to the augmented resistance to local strain softening, caused by inhibition of dislocation motion by Al3(Sc,Zr) dispersoids and subgrain boundaries. In addition, a high density of Al3(Sc,Zr) dispersoids can strongly trap vacancies, which is responsible for the increase in critical strain.