Thanks to their excellent properties such as superelasticity, high hardness, and shape memory effect, polycrystalline shape memory alloys (SMAs) have extensive applications in various engineering fields including automobile, functional materials, and aerospace. Using molecular dynamics simulations, the present paper aims to a systematic study of the fundamental tensile behavior in the nanoscale of polycrystalline B2-CuZr SMAs with mean grain sizes in the range of 6–25 nm. Effects of mean grain size, temperature, and tensile rate on mechanical properties are considered. Our results show that along with the increase in mean grain size came increases in Young's modulus, yield strength, flow stress, and ultimate tensile strength. The development of amorphous regions in the grain cores is the major deformation mode in polycrystalline CuZr SMAs with larger grain sizes, while the grain boundary sliding and grain rotation for smaller grain sizes. Besides, an increased temperature results in mechanical performance degradation and the temperature sensitivity of mechanical properties does not depend on the mean grain size. Our work would lay the groundwork for the optimization of the mechanical properties of polycrystalline SMAs as well as serving as a useful theoretical guideline for their practical engineering applications.