Steel slag pervious concrete (SSPC) is a promising sustainable pavement material for sponge city construction. The influence of aggregate size and notch depth ratio on fracture performance of SSPC was studied in this work. Considering two kinds of aggregate size (4–6 mm and 10–20 mm) and four kinds of notch depth ratio (0.2, 0.3, 0.4, 0.5), the three-point bending test of a rectangular beam with a single side notch was carried out, and the whole fracture process was monitored by acoustic emission. The experimental results show that the fracture behavior of SSPC has obvious boundary effect. Meanwhile, large-size aggregates can improve the fracture strength and fracture energy of SSPC, which is related to the stronger mechanical interlocking and bonding between aggregates caused by thicker slurry layer on the surface of large-size aggregates. In addition, the acoustic emission monitoring results show that the internal wave source activity and strength are higher in the fracture process of large-size aggregate specimens, and there are more shear cracks. Finally, the PFC3D numerical model established by the discrete element theory was used to simulate the fracture behavior of SSPC. The results show that the particle flow code is an effective method to study the fracture behavior of pervious concrete.