In this paper, several kinds of defects induced by post-growth annealing were studied. The type, morphology, origin and formation mechanism of these defects were analyzed and discussed. The defects were characterized by a number of testing methods. The results indicated that high temperature (>973 K) and long time (240 h) annealing were harmful to CdZnTe and CdMnTe crystals, which led to thermal etch pit and surface and internal oxidation. The phenomena of thermal migration of Te inclusions and integration of precipitate/inclusion were observed during isothermal annealing with lower temperature (773–873 K) and shorter time (15–60 h). Stress, atmosphere and temperature gradient annealing were main factors for induced defects such as crack, warping, bulge and Zn-related defect. In order to avoid these defects, suitable annealing source and atmosphere for crystals of different properties, high vacuum annealing quartz tube, optimized annealing temperature and time, smaller temperature gradient and multiple annealing methods were proposed. The research would be helpful for crystal growth and post-growth annealing of CdTe-based semiconductor materials.