The aggregate in the ballast layer of the railway track structure degrades due to the splitting of single particles and the abrasion of the angular corners, resulting in a ballast course comprised of degraded aggregate with a lower drainage potential. In the present study, the hydraulic conductivity of clean, degraded and clay-fouled degraded ballast specimens was evaluated considering the initial particle size distribution of aggregate and the degradation mechanism to investigate the permeability of ballast fouled by both internal and external factors. First, degraded ballast aggregate was generated by using a large-scale impact loading testing apparatus. Then, a large-scale constant head permeability test was performed to estimate the hydraulic conductivity of degraded ballast aggregate contaminated with clay. Based on the obtained results, the flow regime through extremely degraded ballast remained between a completely turbulent regime (observed in the case of clean ballast) and a laminar flow regime. Darcy’s law was shown to be valid for degraded aggregate samples contaminated with clay. While the degradation of the aggregate had the expected effect of reducing hydraulic conductivity, the abrasion of asperities was shown to have an even greater impact on hydraulic conductivity. Also, the further degradation of aggregate under impact loading was shown to accelerate the reduction in the permeability of clay-fouled degraded ballast to an unacceptable extent.