Bicarbonate (HCO3‐) is an essential factor to modulate sperm motility. Female genital tract fluids are rich in HCO3‐ and contribute to an increase of sperm beat frequency thus supporting sperm to reach the site of fertilization. Carbonic anhydrases (CA) catalyze the reversible hydration of CO2 to HCO3‐ and are important candidates in the regulation of both the HCO3‐ homeostasis in sperm and the composition of the male and female genital tract fluids. We show that CAII and CAIV are localized in epithelial cells lining the epididymal duct and that they appear with the onset of puberty. Gene expression analyses with tissue from respective knockout animals reveal an up‐regulation of CAII and CAIV in the different epididymal sections, indicating a compensatory mechanism for the gene defect. In sperm, the presence of CAII is restricted to the principal piece, whereas CAIV is detectable in the plasma membrane of the entire sperm tail. CAII and CAIV single knockout animals display an impaired HCO3‐ homeostasis resulting in significantly diminished sperm motility, velocity and response to HCO3‐. Enzyme activity tests show that CAII and CAIV account for nearly 95% of the total CA activity in sperm. We generated CAII/CAIV double knockout animals which display a significant reduction in sperm motility resulting in sub‐fertility. Interestingly, biochemically created CAIV deficient sperm of CAII knockout animals show a lower response to HCO3‐ as compared to sperm from double knockout animals. From these results we conclude that both CAII and CAIV play a key role for a successful fertilization.