Background: This study evaluated the feasibility of reducing the radiation dose in abdominal imaging of urolithiasis with a clinical photon-counting CT (PCCT) by gradually lowering the image quality level (IQL) without compromising the image quality and diagnostic value. Methods: Ninety-eight PCCT examinations using either IQL70 (n = 31), IQL60 (n = 31) or IQL50 (n = 36) were retrospectively included. Parameters for the radiation dose and the quantitative image quality were analyzed. Qualitative image quality, presence of urolithiasis and diagnostic confidence were rated. Results: Lowering the IQL from 70 to 50 led to a significant decrease (22.8%) in the size-specific dose estimate (SSDE, IQL70 4.57 ± 0.84 mGy, IQL50 3.53 ± 0.70 mGy, p < 0.001). Simultaneously, lowering the IQL led to a minimal deterioration of the quantitative quality, e.g., image noise increased from 9.13 ± 1.99 (IQL70) to 9.91 ± 1.77 (IQL50, p = 0.248). Radiologists did not notice major changes in the image quality throughout the IQLs. Detection rates of urolithiasis (91.3-100%) did not differ markedly. Diagnostic confidence was high and not influenced by the IQL. Conclusions: Adjusting the PCCT scan protocol by lowering the IQL can significantly reduce the radiation dose without significant impairment of the image quality. The detection rate and diagnostic confidence are not impaired by using an ultra-low-dose PCCT scan protocol.