Polarization effect has become one of the main factors that limit the application of CdZnTe X-ray photon-counting detectors. In this study, we systematically investigate the impact of deep-level traps on the carrier transport properties and explore its underlying correlation with the polarization effect. Besides, the critical deep-level traps affecting photon counting performance are identified. Two types of CdZnTe detectors with significant differences in counting performance are explicitly selected for comparative research. The carrier transport characteristics are discussed by analyzing X-ray and γ-ray energy spectrum responses. The data shows the detectors with larger (μτ)h and smaller (μτ)e have better photon counting performance. This type of detectors can withstand higher X-ray doses without polarization. However, their relatively poor electron transport performance will reduce the detection efficiency. We comprehensively analyzed the TSC and i-DLTS results to obtain deep-level trap characteristics of the two types of CdZnTe detectors. The results indicate that CdZnTe detectors with good hole transport properties and photon counting performance have fewer Tei, Te secondary phases and Te precipitates related defects identified as deep hole traps with large capture cross-sections. Their larger Tecd2+ concentration, which acts as a deep electron trap, should be responsible for poor electron transport properties and consequently reduced detection efficiency.