To optimize the pore skeleton structure of pervious concrete to enhance its phosphorus (P) removal from stormwater, small-sized adsorptive aggregate (1–2 mm steel slag) and large-sized natural aggregate (2–5 mm basalt), are applied for the manufacture of two-sized aggregate pervious concrete. Physico-mechanical properties, adsorption performance and freeze–thaw resistance are investigated. The results show that the small-sized steel slag aggregate fills the pores between the large-sized natural aggregates and reduces the porosity and permeability of pervious concrete, significantly improving the mechanical strength and freeze–thaw resistance. The pervious concrete in this study shows an excellent P-adsorption capacity, all P is removed from the aqueous solution with an initial concentration of 168 mg/L and 307 mg/L. The harmful elements (Cr, Sr and V, etc.) leached from the concrete are lower than the maximum limit value. The orthogonal results show that cement and steel slag are the main factors affecting the P-adsorption performance of pervious concrete. Based on the current results, sample 3 (cement: 350 kg/m3, sand: 500 kg/m3, W/C: 0.37, 1–2 mm steel slag: 1385.8 kg/m3 and 2–5 mm basalt: 361.3 kg/m3) shows good P-removal capacity, which is an optimal mix for pervious concrete to remove P from stormwater, with suitable strength and durability.