Purpose: Developing photon-counting CT detectors requires understanding the impact of parameters, such as converter material, thickness, and pixel size. We apply a linear-systems framework, incorporating spatial and energy resolution, to study realistic silicon (Si) and cadmium telluride (CdTe) detectors at a low count rate. Approach: We compared CdTe detector designs with and pixels and Si detector designs with pixels of 30 and 60mm active thickness, with and without tungsten scatter blockers. Monte-Carlo simulations of photon transport were used together with Gaussian charge sharing models fitted to published data. Results: For detection in a 300-mm-thick object at 120kVp, the 0.5- and 0.225-mm pixel CdTe systems have 28% to 41% and 5% to 29% higher detective quantum efficiency (DQE), respectively, than the 60-mm Si system with tungsten, whereas the corresponding numbers for two-material decomposition are 2% lower to 11% higher DQE and 31% to 54% lower DQE compared to Si. We also show that combining these detectors with dual-spectrum acquisition is beneficial. Conclusions: In the low-count-rate regime, CdTe detector systems outperform the Si systems for detection tasks, whereas silicon outperforms one or both of the CdTe systems for material decomposition.