A novel dislocation mechanism to interpret the Bauschinger effect in polycrystalline metals is proposed namely dislocation pile-up polarization. We show that the variability of GB resistance to dislocation transmission, an inherent characteristic in polycrystalline metals, can break the symmetry of double dislocation pile-ups during plastic deformation, resulting in the polarization of dislocation pile-ups. This polarization mechanism strengthens the material in forward loading but softens it in reverse loading, thus being a source of back stress, which is verified by the dislocation dynamic simulations. Further, an analytical model based on the polarization mechanism is proposed and coupled with the crystal plasticity constitutive framework to analyze the stress–strain behavior of materials during loading and reverse loading. It is found that the Bauschinger effect and the strain-dependent Hall–Petch effect naturally appear as the intrinsic properties of polycrystalline metals in this theoretical framework, and the associated microstructural sensitivity is well captured. Our theoretical studies demonstrate the key role of GB resistance variability in regulating mechanical properties of polycrystalline metals and shed light on the physical mechanism of the Bauschinger effect. Moreover, this study indicates that the back stress hardening can be enhanced by the polarization mechanism, which might reveal new routes for design of high-performance metallic materials.