We conducted pot experiments to assess the bioavailability of cadmium (Cd) in contaminated rhizosphere soil and accumulation in rice organs in response to nitrogen (N) supply ((NH4)2SO4, NH4NO3, NH4Cl). The results showed that the concentration of bioavailable Cd in rice rhizosphere soil was (NH4)2SO4 treatment > NH4Cl treatment > NH4NO3 treatment at the same level of N application and growth period; the Cd concentration in rice roots was (NH4)2SO4 treatment > NH4NO3 treatment > NH4Cl treatment; and the Cd concentration in rice straw was NH4NO3 treatment > NH4Cl. The Cd concentration in rice roots, straws, and seeds at the maturity stage was (NH4)2SO4 treatment > NH4Cl treatment. With the same N fertilizer, excessive N promoted Cd accumulation in rice at later growth stages. This suggested that sulfate (SO42−) influenced Cd concentration in rice. NH4Cl application maintained a low Cd level in different rice organs with the same N level. This confirmed that NH4Cl is a safe N source for rice planting in polymetallic sulfide mining areas. The study concludes that appropriate NH4Cl levels for Cd-contaminated paddy soil with high-S-content could obtain rice grains with Cd concentrations below the food safety standards (0.2 or 0.4 mg·kg−1).