In the context of polycrystalline metals, the damage process analysis is generally restricted to surface of a sample containing some hundreds of grains, due to the inherent difficulties microstructural analysis. Determination of the grain size influence, misorientation and neighboring grains effect remains difficult with experimental studies. In this work, the influence of microstructural characteristics of polycrystalline metals on the evolution of the damage process under cyclic loading is investigated. On the basis of the quaternion theory, the orientation of each (crystal) grain is represented by a single angle (theta). The relative misorientation of each grain is set by the ratio of its misorientation over the average value of polycrystalline aggregate. The relative volume of grain is used as a parameter representing the grain size. The damage evolution under cyclic loading is identified using the coupling of the crystal plasticity model with the Continuum Damage Mechanics (CDM) model. The damage evolution and its distribution on polycrystalline aggregate are calculated by means of numerical homogenization over each grain. The heterogeneity distribution and damage evolution for polycrystalline metals have been analyzed. The obtained results show that neighboring grains effects are larger and may change the tendency that larger grains with great misorientation are the most damaged.